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Abstract 

Satellite  constellation  resiliency  is  an  important  consideration  gaining  momentum 
at  the  top  levels  of  the  Air  Force  and  at  Air  Force  Space  Command  (AFSPC).  The 
increased  availability  of  threats  to  satellite  systems  is  challenging  the  capabilities 
provided  by  space  assets.  More  specifically,  the  global  positioning  system  (GPS) 
satellite  constellation  is  utilized  for  a  variety  of  missions,  to  include  providing  precise 
geolocation  information  for  navigation.  Any  degrade  in  GPS  capabilities  as  observed 
in  an  urban  canyon  environment  or  due  to  the  loss  of  a  GPS  satellite  may  hinder 
the  overall  mission.  We  use  the  System  Effectiveness  Analysis  Simulation  (SEAS) 
to  model  the  GPS  constellation  in  an  urban  canyon  environment  which  provides 
information  to  a  special  operation  force  (SOF)  in  their  effort  to  recover  a  weapon 
of  mass  destruction  (WMD).  By  varying  the  type  of  operations  and  the  number 
of  satellites  lost  in  the  simulation,  insight  is  gained  into  the  impact  of  degradation 
through  the  selected  top  level  mission  metrics.  A  series  of  statistical  difference  tests 
and  a  designed  experiment  reveal  a  resiliency  threshold  on  the  number  of  satellites 
removed  from  the  constellation.  As  a  result,  we  conclude  that  the  GPS  constellation 
is  resilient  even  after  the  loss  of  several  satellites. 

Key  words:  System  Effectiveness  Analysis  Simulation  (SEAS),  satellite  constella¬ 
tion  resiliency,  agent-based  modeling,  global  positioning  system  (GPS),  urban  canyon 
environment,  weapon  of  mass  destruction  (WMD) 
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ASSESSING  GPS  CONSTELLATION  RESILIENCY  IN  AN  URBAN  CANYON 


ENVIRONMENT 

I.  INTRODUCTION 


1.1  Background 

Satellite  design  is  shifting  from  large  monolithic  satellites  of  the  cold  war  to  smaller 
more  disaggregated  satellite  constellations.  The  change  is  prompted  by  increasingly 
accessible  satellite  degradation  techniques  which  may  inhibit  national  security  by 
reducing  the  support  of  mission  critical  space  systems.  Butler  [8]  states  that,  “All  of 
the  Pentagon’s  war  plans  are  heavily  reliant  on  satellite  service,  and  the  economies 
of  the  U.S.  [United  States]  and  its  allies  also  depend  on  spaceborne  services  such  as 
GPS  [Global  Positioning  System]  and  communications  for  smooth  operation”.  Air 
Force  Space  Command  [3]  (AFSPC)  also  states  that,  “U.S.  reliance  on  space  was  a 
potential  Achilles  Heel” .  The  vulnerability  in  our  space  systems  is  due  to  an  inherent 
susceptibility  to  a  variety  of  threats  as  noted  in  the  following  statement  by  Dr.  Stuart 
Eves:  “[a]  spectrum  of  threats  from  [ASAT]  weapons,  RF  weapons,  cyber  attacks, 
demons  conducting  disruption  or  surveillance  operations,  physical  attack  on  ground 
infrastructure,  laser  weapons,  charged  and  neutral  particle  beams,  and  camouflage 
concealment  and  deception”  [7].  U.S.  satellite  systems  must  be  defended  from  these 
threats  to  ensure  national  security. 

Acting  Air  Force  secretary  Eric  Fanning  also  recognizes  the  vulnerability  of  our 
military  space  systems  [20].  Interestingly,  he  notes  that  we  do  not  necessarily  need 
an  offensive  capability  to  defend  our  space  assets  [20].  Overall,  he  asserts  the  need 
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for,  “new  strategies  and  new  architectures  for  space  to  try  to  increase  resilience”  [20] . 
General  Shelton,  former  commander  of  AFSPC,  echoes  Mr.  Fanning’s  request  for 
an  increase  in  satellite  constellation  resiliency  [3].  As  a  result,  the  Space  and  Mis¬ 
sile  Center  (SMC)  of  the  United  State  Air  Force  (USAF)  has  conducted  a  Resilient 
Enterprise  Architecture  Pathfinder  (REAP)  study,  a  leading  project  in  space  system 
resiliency  analysis.  The  results  from  the  REAP  study  and  other  resiliency  research 
will  have  far  reaching  implications  that  will  effect  the  design  and  acquisition  process  of 
future  satellite  space  systems.  Ensuring  the  security  and  capability  of  military  satel¬ 
lite  constellations  through  increasing  efforts  toward  resiliency  is  essential  for  national 
security. 

1.2  Problem  Definition 

The  focus  of  this  thesis  is  on  the  comparison  of  resiliency  metrics  for  the  GPS 
constellation  under  different  levels  of  degradation.  Resiliency  is  difficult  to  define 
due  to  its  applicability  in  many  areas  of  study.  It  is  even  more  difficult  to  find  a 
quantitative  measurement  to  use  as  a  standardized  metric  for  comparing  the  resilience 
of  different  systems.  Measures  of  performance  (MOPs),  measures  of  effectiveness 
(MOEs),  and  measures  of  outcome  (MOOs)  are  all  common  quantitative  metrics 
available  for  a  system.  Selecting  appropriate  and  universally  accepted  metrics  is 
essential  for  credible  analysis. 

1.3  Research  Objective 

The  goal  of  this  thesis  is  to  successfully  compare  the  resiliency  of  the  GPS  satellite 
constellation  in  an  urban  canyon  environment  using  quantitative  metrics  for  a  well 
defined  scenario.  The  metrics  selected  reflect  overall  top  level  mission  priorities, 
also  known  as  MOOs.  The  MOOs  are  primarily  drawn  from  the  suggested  measures 
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found  in  the  Literature  Review  along  with  the  capabilities  of  the  System  Effectiveness 
Analysis  Simulation  (SEAS)  package.  The  thesis  captures  the  simulation  of  the  GPS 
satellite  constellation  interacting  with  a  blue  force  in  a  blue  and  red  force  conflict 
scenario.  MOOs  are  collected  for  a  nominal  scenario  as  well  as  for  varying  levels 
of  degraded  performance  representing  system  failures,  environmental  challenges,  or 
adversarial  actions. 

1.4  Research  Scope 

Simulation  in  this  thesis  is  limited  to  a  SEAS  scenario  provided  by  SMC.  The 
simulation  models  each  of  the  satellites  in  the  constellation  as  unique  agents  that  can 
be  impacted  by  threats.  Additionally,  the  thesis  only  compares  the  top  level  mission 
focused  MOOs  instead  of  focusing  on  the  low  level  technical  MOEs  and  MOPs  of  the 
system.  The  analysis  of  the  simulation  MOOs  includes  statistical  Paired  T  tests  and 
proportion  confidence  interval  comparisons  between  nominal  and  degraded  scenarios 
to  provide  insight  into  the  system.  Furthermore,  a  full  factorial  designed  experiment 
is  performed  to  determine  the  most  important  factors  for  the  selected  MOOs.  The 
designed  experiment  has  two  factors  with  multiple  levels  which  include  the  type  of 
scenario,  either  nominal  or  degraded,  and  the  number  of  satellites  removed  from  the 
constellation  from  zero  to  ten.  Both  methods  of  analysis  provide  insight  into  the 
resiliency  of  the  GPS  satellite  constellation. 

1.5  Thesis  Overview 

This  thesis  contains  five  chapters,  the  Introduction,  Literature  Review,  Method¬ 
ology,  Analysis,  and  Conclusions.  The  Literature  Review  covers  the  definition  of 
resiliency  and  its  associated  metrics,  an  overview  of  the  threats  to  satellite  systems, 
a  description  of  the  GPS  constellation,  and  information  on  SEAS.  The  Methodology 
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provides  framework  for  the  simulation  and  the  scenario  development.  Next,  the  Anal¬ 
ysis  chapter  focuses  on  the  impact  of  the  input  parameters  on  the  GPS  constellation 
and  the  insights  gained  on  the  GPS  constellation’s  resiliency.  The  Conclusion  chapter 
summarizes  the  important  insights  gained  from  the  thesis  along  with  identifying  areas 
for  further  research. 
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II.  LITERATURE  REVIEW 


2.1  Overview 

The  Literature  Review  focuses  on  several  areas  which  support  resiliency  analysis. 
The  first  section  identifies  the  applications  of  resiliency  across  many  different  areas  of 
study  and  selects  a  definition  for  space  resiliency.  Additionally,  the  section  identifies 
current  investigations  into  satellite  resiliency  along  with  potential  metrics  for  the 
thesis.  Second,  the  Literature  Review  examines  threats  to  satellite  systems  and  their 
associated  countermeasures,  if  applicable.  Third,  the  Literature  Review  provides  an 
overview  of  the  GPS  satellite  constellation  and  its  capabilities.  Fourth,  the  Literature 
Review  describes  agent  based  modeling  and  its  implementation  in  SEAS. 

2.2  Resiliency 

Applications. 

There  are  an  array  of  different  definitions  and  applications  for  resiliency  rang¬ 
ing  from  ecology  to  economics  [30].  One  example  is  the  Environmental  Protection 
Agency  (EPA)  which  recognizes  that  resiliency  analysis  is  still  developing  and  does 
not  have  standardized  metrics  [16].  Nevertheless,  the  EPA  does  see  the  applicabil¬ 
ity  of  resiliency  studies  to  better  understand  social-ecological  systems  [16].  The  EPA 
also  references  an  organization  titled  the  Resilience  Alliance  which  specifically  focuses 
on  social-ecological  systems  [31].  Additionally,  Ohio  State  University  has  a  Center 
for  Resilience  that  examines  the  resilience  of  industrial  systems  and  their  operation 
[10].  Furthermore,  research  extends  to  the  MITRE  corporation  which  provides  an 
extensive  list  of  resiliency  metrics  for  the  cyber  domain  [6].  As  shown  by  the  vari¬ 
ety  of  different  resiliency  applications,  Reid  &  Botterill  [30]  argue  that  ‘resiliency’  is 
becoming  convoluted  and  losing  it  meaning  due  to  improper  and  over  use.  To  add 


5 


to  the  convolution  Reid  &  Botterill  [30]  note  that  resiliency  can  often  be  mistaken 
as  ‘recovery’  or  ‘adaptability’  or  ‘vulnerability’.  The  examples  provided  are  a  small 
portion  of  the  numerous  applications  of  resiliency. 

A  specific  example  of  space  resiliency  comes  from  Buckerheld  de  la  Roche  [7]  who 
comments  on  the  events  of  the  2011  RUSI  Space  and  National  UK  Security  conference. 
The  conference  was  focused  on  increasing  resiliency,  which  included  a  discussion  on 
cyber  and  military  actions  in  the  space  domain.  Space  system  resilience  has  also  taken 
hold  in  the  United  States  where  General  Shelton,  former  Air  Force  Space  Command 
commander,  provided  the  following  statement: 

Our  satellites  provide  a  strategic  advantage  for  the  U.S.,  and  as  such, 
we  must  consider  the  vulnerabilities  and  resilience  of  our  constellations. 

My  staff  at  headquarters  Air  Force  Space  Command,  alongside  the  team 
at  the  Space  and  Missile  Systems  Center,  is  leading  efforts  at  balancing 
resilience  with  affordability.  [3] 

Resiliency  in  space  systems  is  important  to  overall  military  success  and  must  be 
defined  for  the  proper  application. 

Definition. 

Air  Force  Space  Command  provides  a  definition  of  resiliency:  “Resiliency  is  the 
ability  of  a  system  architecture  to  continue  providing  required  capabilities  in  the 
face  of  system  failures,  environmental  challenges,  or  adversary  actions”  [3].  We  use 
AFSPC’s  definition  of  resiliency  for  the  remainder  of  the  thesis.  The  definition  is 
unique  because  it  encompasses  other  common  resiliency  definitions.  For  example,  the 
adaptability  of  a  space  system  to  threats  may  be  a  method  for  maintaining  required 
capabilities.  Additionally,  the  definition  speaks  to  the  system  architecture  as  a  whole 
instead  of  just  specific  portions  of  the  system.  The  all  encompassing  definition  allows 
for  resiliency  to  be  applied  not  only  to  the  satellite  constellation,  but  also  to  the 
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ground  station  and  satellite  operators  who  may  be  effected  by  threats  to  overall 
resiliency. 

Current  Research. 

Investigation  into  space  system  resiliency  is  a  new  area  of  study  being  explored  by 
a  number  of  researchers  including  Northrop  Grumman  Aerospace  Systems.  Northrop 
Grumman  argues  that  space  system  resiliency  is  best  assessed  by  the  ability  of  the 
system  to  meet  key  performance  parameters  (KPPs).  Northrop  Grumman  also  iden¬ 
tifies  two  different  KPP  approaches  to  assessing  satellite  resiliency  under  adversarial 
threats:  analytical  and  deterministic  modeling.  Analytical  modeling  aims  to  create 
a  mathematical  function  to  objectively  calculate  resiliency.  Then  by  comparing  the 
magnitude  of  the  resiliency  function  for  different  systems,  the  most  resilient  system 
can  be  selected.  Northrop  Grumman  Aerospace  Systems  indicates  that  an  analytical 
model  is  difficult  to  create  due  to  different  opinions  on  the  proper  function  to  select  for 
resiliency  measurement.  Deterministic  modeling  instead  focuses  on  the  breakpoints 
of  the  space  system.  Furthermore,  in  a  deterministic  model  it  appears  as  though 
cost  and  reliability  become  metrics  for  comparison.  Essentially,  the  deterministic 
model  compares  the  cost  of  the  military  space  system  as  a  ratio  to  the  cost  of  the 
threat.  Comparing  the  ratio  to  possible  alternative  space  systems  helps  determine 
which  system  is  more  resilient.  Northrop  Grumman  Aerospace  Systems  argues  that 
a  deterministic  model  is  most  effective  at  identifying  large  differences  in  resiliency. 
Intuitively,  Northrop  Grumman  Aerospace  Systems  notes  from  initial  results  that 
space  systems  that  can  be  impacted  from  ground  attacks  (whether  through  a  ground 
station  or  from  ground  techniques)  are  less  resilient.  [26] 

A  more  specihc  GPS  constellation  analysis  thesis  was  produced  in  2010  by  Cap¬ 
tain  Bryan  Bell  at  the  Air  Force  Institute  of  Technology  (AFIT)  titled,  “Assuming 
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GPS  Capabilities  Under  A  Contested  Space  Environment:  An  Implementation  Plan”. 
Bell’s  research  focuses  specifically  on  how  best  to  augment  the  current  GPS  constella¬ 
tion  to  retained  its  performance  under  degradation;  thereby  fulfilling  the  requirements 
of  Operationally  Responsive  Space  (ORS).  For  educational  purposes  only,  Bell’s  area 
uses  Taipei,  Taiwan  as  his  area  of  interest.  His  GPS  constellation  is  modeled  in  the 
Satellite  Toolkit  (STK)  where  augmentation  satellites  can  be  added  and  the  mission 
level  technical  metrics  can  be  recorded.  The  principle  metric  Bell  uses  is  the  position 
dilution  of  precision  (PDOP),  which  is  a  common  technical  metric  detailing  the  ge- 
olocation  geometry  provided  by  the  GPS  constellation.  The  goal  of  his  thesis  is  to 
identify  degraded  scenarios  where  the  average  and  maximum  PDOP  exceed  a  value 
of  six  often  enough  to  impact  military  operations  and  then  improve  those  situations 
with  satellites  to  augment  the  constellation.  He  uses  a  variety  of  different  satellite 
augmentations  to  include  the  low  Earth  orbit  (LEO)  iridium-66  constellation,  a  highly 
eccentric  orbit  (HEO)  constellation,  and  a  geosynchronous  Earth  orbit  (GEO)  con¬ 
stellation.  Bell  concludes  that  the  GPS  constellation  is  a  robust  system  that  can  be 
best  augmented  by  adding  several  satellites  in  GEO.  One  important  insight  gained 
from  the  study  is  the  importance  of  the  geometry  of  the  satellites  over  the  area  of 
interest.  He  notes  that  scenarios  with  more  satellites  overhead  does  not  necessarily 
mean  that  the  PDOP  value  will  be  lower  than  a  constellation  where  fewer  satellites 
provide  better  geometry.  While  Bell’s  research  is  important  to  consider  moving  for¬ 
ward  in  the  methodology  of  our  study,  the  research  presented  in  our  study  selects 
different  metrics  and  methods  for  analyzing  satellite  constellation  resiliency.  [5] 

Available  Metrics. 

Selecting  the  correct  metrics  is  crucially  important  to  the  validity  of  the  final 
anaylsis.  The  selected  metrics  are  also  important  because  they  define  what  is  relc- 


vant  to  the  decision  maker.  Misinterpretation  of  the  metrics  could  also  occur  without 
a  universal  acceptance  of  the  metrics.  Furthermore,  the  selected  metrics  should  be 
parsimonious,  allowing  for  flexibility  and  application  to  multiple  scenarios.  More 
specifically,  Connable  references  David  Kilcullen’s  opinion  on  the  metrics  used  during 
the  war  in  Afghanistan.  Kilcullen  believes  that  using  a  single  number  as  a  metric, 
such  as  the  number  of  enemy  casualties,  can  be  dangerous  when  taken  out  of  context. 
Instead,  Kilcullen  recommends  using  ratios  or  percentages  which  can  be  more  infor¬ 
mative  metrics.  For  example,  the  hre-to-fire  ratio  allows  an  analyst  to  observe  which 
side  has  the  initiative  and  is  more  likely  in  control  of  the  casualty  rate.  [12] 

There  are  several  quantitative  metrics  available  for  analyzing  satellite  system  re¬ 
siliency.  For  example,  Cornara  et  al.  [13]  states  that  all  satellite  constellations  aim 
to  provide  a  minimum  level  of  service.  Bell  believes  that  a  PDOP  value  of  six  defines 
minimum  GPS  service  [5].  Cornara  et  al.  [13]  also  suggests  using  mean  revisit  time 
and  mean  percentage  of  coverage  during  the  simulation  as  measures  of  quality  of  the 
constellation.  The  REAP  study  focuses  on  MOEs  and  uses  availability,  user  range 
error,  vertical  error,  horizontal  error,  and  integrity  as  metrics  [22],  The  REAP  study 
also  references  several  MOOs  to  include  target  production  rate,  target  strike  accu¬ 
racy,  total  targets  destroyed,  and  days  to  end  of  conflict  [22],  Other  potential  MOOs 
include  the  accuracy  of  weapons,  the  amount  of  required  ordinance,  the  number  of 
civilian  casualties,  and  the  amount  of  fuel  expended  by  the  satellite.  It  is  also  im¬ 
portant  to  remember  that  the  selected  metrics  must  also  only  be  calculated  over  the 
area  of  interest.  The  selected  area  might  be  a  specific  geographic  region  or  a  more 
general  bound  of  latitude  and  longitude. 
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Improvements  to  Constellations. 


There  are  several  considerations  for  enhancing  satellite  constellation  resiliency. 
The  first  is  that  a  constellation  with  a  smaller  number  of  orbital  planes  reduces  the 
impact  of  satellite  degradation  and  also  achieves  levels  of  service  earlier  in  the  deploy¬ 
ment  phases.  The  second  consideration  is  the  possibility  of  having  reserve  satellites 
ready  for  operation  in  a  degraded  environment.  Cornara  et  al.  [13]  references  three 
general  options  for  replacement  satellites.  The  best  option  is  to  have  additional  satel¬ 
lites  in  the  orbital  planes  ready  for  operation.  This  option  is  generally  implemented 
for  position,  navigation,  and  timing  (PNT)  constellations  or  any  other  constella¬ 
tions  which  demand  continuous  functionality.  The  second  most  reactive  method  is 
to  have  spare  satellites  in  parking  orbits  which  can  be  transferred  into  operation 
when  required.  The  final  option,  generally  used  by  less  essential  systems  such  as 
Earth  observation  platforms,  is  to  have  back  up  satellites  on  the  ground  waiting  for 
launch.  Depending  upon  the  objective  of  the  satellite  system  and  its  overall  mission 
importance,  the  correct  resiliency  method  can  be  implemented.  [13] 

2.3  Threats 

Military  satellite  systems  are  crucial  to  the  U.S.  however,  they  are  also  highly 
vulnerable.  Northrop  Grumman  Aerospace  Systems  [26]  states  that,  “especially  trou¬ 
bling  are  the  low  cost  and  short  cycle  times  of  very  effective  threats  when  compared 
with  the  investments  that  are  made  in  [Department  of  Defense]  space  systems” .  Airst 
[4]  also  states  that,  “[t]hreat[s]  [have]  expanded  from  nation-state  to  informed  hobbi- 
est” .  The  escalation  of  the  availability  of  threats  only  increases  the  need  for  research 
into  space  system  resiliency. 
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Foreign  Countries. 


China  has  recently  demonstrated  the  ability  to  destroy  a  satellite  in  LEO  and 
potentially  degrade  optical  payloads  with  lazer  dazzling  [3].  Pawlikowski  et  al.  [28] 
recognizes  that  China  is  attempting  to  weaken  the  United  States,  who  is  increasingly 
reliant  on  their  space  capabilities.  The  Chinese  launch  of  an  anti-satellite  (ASAT) 
missile  was  an  attack  the  Linked  States  could  only  observe  and  not  take  action  against 
[28].  Additionally,  Pawlikowski  et  al.  [28]  states  that  China  is  not  the  only  concern, 
other  states  and  even  non-state  actors  could  also  impact  space  systems.  As  foreign 
countries  develop  their  technology  and  become  more  capable,  the  Linked  States  will 
have  to  protect  the  GPS  constellation  from  jamming,  spoofing,  cyber,  and  kinetic 
threats.  [28] 

Jamming. 

Jamming  is  a  common  and  easily  implementable  threat  available  to  most  foreign 
actors.  Pawlikowski  et  al.  [28]  states, 


The  [technological  capability  to  jam  satellites  is  fairly  simple  and  can 
be  easily  assembled  by  either  individuals  or  nations  for  a  fairly  modest 
investment.  Multiple  reports  of  both  state  and  nonstate  groups  jamming 
satellites  have  been  seen  over  the  last  decade.  GPS  jammers  are  well 
known  and  offered  openly  for  sale  on  the  Internet.  Satellite  transit  times 
are  available  from  several  websites  and  can  be  downloaded  onto  smart 
phones. 

Additionally,  jamming  a  GPS  antennae  does  not  require  more  than  a  few  picowatts 
of  energy  [4],  As  a  result,  foreign  militaries  have  jammers  that  can  output  megawatts 
of  power  which  are  highly  effective  [4],  Furthermore,  jamming  can  occur  through 
attacks  on  the  narrow  band,  broadband,  and  spread  spectrum  [4], 
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More  specifically,  North  Korea’s  usage  of  jammers  since  2003  during  the  Iraq  war 
has  been  confirmed,  along  with  reports  of  China  also  having  jammers  [4],  Iran  also 
claims  to  have  taken  control  of  a  lost  U.S.  stealth  drone  in  2011  by  jamming  its  GPS 
connection  [9].  An  Iranian  engineer  claims  that,  “The  GPS  navigation  is  the  weakest 
point. ..By  putting  noise  [jamming]  on  the  communications,  you  force  the  bird  into 
autopilot.  This  is  where  the  bird  loses  its  brain”  [9].  General  Moharam  Gholizadeh 
of  the  Iranian  Islamic  Revolutionary  Guard  Corps  believes  that  Iran  can  even  control 
GPS-guided  missiles  [9].  Furthermore,  even  civilians  have  access  to  jammers  in  order 
to  ward  off  vehicle  tracking  devices  [4], 

The  DoD  is  aware  of  the  jamming  threat  and,  as  a  result,  the  new  GPS  satellites 
will  have  increased  capability  to  prevent  jamming  [11],  There  are  also  several  defensive 
techniques  to  combat  jamming.  Inherent  in  a  GPS  satellite  is,  “[t]he  ability  to  reject 
noise  [which]  also  impliefs]  a  powerful  ability  to  reject  most  forms  of  jamming  or 
deliberate  interference”  [27].  The  military  also  uses  the  p-code  from  the  GPS  satellite 
which  is  more  resistant  to  jamming  [27].  Nevertheless,  jamming  is  still  a  potential 
threat  to  GPS  satellites,  which  can  degrade  overall  mission  effectiveness. 

Spoofing. 

Spoofing  is  another  type  of  attack  which  is  applied  to  GPS  receivers  [21],  By  sim¬ 
ply  adding  a  time  delay  to  a  GPS  signal,  adversaries  can  “spoof”  or  confuse  a  GPS 
receiver  adding  errors  to  position  estimations  and  providing  incorrect  time  stamps  [4] . 
Errors  in  position  estimates  can  be  devastating  for  the  military  technology  which  aims 
to  hit  targets  with  precision  in  order  to  use  a  minimal  amount  of  ordinance.  Further¬ 
more,  the  probability  of  civilian  casualties  also  increases  if  an  ordinance  receives  even 
the  slightest  error  measurement  from  its  guidance  system.  Another  example  from 
Airst  [4]  centers  aounrd  the  stock  market  which  is  heavily  dependent  upon  the  time 
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information  provided  by  GPS  satellites.  If  a  company  can  control  the  time  stamp  for 
trades  in  the  market,  they  can  easily  manipulate  stock  trading  in  order  to  sell  high 
and  buy  low  [4], 

Airst  [4]  argues  that  spoofing  is  a  significant  threat  to  GPS,  primarily  due  to  its 
inexpensive  cost,  quick  availability,  and  hidden  implementation.  There  are  several 
devices  which  are  readily  available  that  can  spoof  GPS  receivers.  A  GPS  simulator 
is  the  most  simplistic  option,  however,  its  large  size  and  expense  makes  it  difficult  to 
implement  [21],  The  next  step  in  complexity  is  the  use  of  a  single  software  defined 
GPS  receiver  to  provide  a  false  GPS  signal  [21],  The  final  step  in  complexity  is  an 
orchestrated  group  of  software  defined  GPS  receivers  [21],  Fortunately,  there  is  still 
additional  software  and  hardware  development  required  before  a  software  defined  GPS 
receiver  can  become  an  effective  spoofing  device  [21],  Additionally,  Airst  [4]  states 
that  there  are  not  any  readily  available  solutions  to  eliminate  spoofing. 

There  are,  however,  several  potential  options  to  minimize  the  impact  of  spoofing 
attacks.  Humphreys  et  al.  [21]  states  that,  “Cryptographic  authentication  is  arguably 
the  most  secure  solution  [to  spoofing],  but  would  require  modification  of  the  civil  GPS 
signal  structure,  making  it  an  unlikely  short-term  solution”.  One  of  the  next  best 
options  for  spoofing  detection  is  to  use  multiple  clocks  in  the  satellite  system  and 
ensure  they  all  are  in  synch  [4],  If  the  clocks  are  disperate,  then  the  GPS  satellite  can 
use  an  “advanced  disciplining  algorithm”  to  place  the  clocks  at  the  correct  time  [4], 
Nevertheless,  more  research  and  development  must  be  performed  to  find  a  simple  yet 
effective  countermeasure  to  spoofing  attacks. 

Cyber. 

Persistent  cyber  threats  also  permeate  into  the  space  domain  due  to  a  satellite’s 
inherent  dependence  upon  computer  technology.  Air  Force  Space  Command  [3]  states 
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that,  “[sjpace  systems  that  rely  on  complex  software  and  radio-frequency  links  could 
be  susceptible  to  [cyber]  attacks,  despite  robust  cryptographic  protection1' .  To  make 
matters  worse,  Air  Force  Space  Command  [3]  states  that,  “[cjyberspace  threats,  in 
particular,  have  exceptionally  low  barriers  to  entry  and  are  growing  rapidly.”  Cyber 
threats  are  so  inexpensive  and  readily  available  that  an  adversary  would  be  ill-advised 
not  to  attempt  such  an  attack  on  the  U.S.  [9]. 

Caplan  [9]  more  specifically  states  that,  “satellite  vulnerability  to  cyber  attacks 
has  emerged  as  a  threat  to  U.S.  national  security,  as  the  U.S.  military  is  increasingly 
dependent  on  satellite  communications” .  It  is  also  important  to  understand  that  the 
military  is  heavily  dependent  upon  civilian  satellites  for  telecommunication.  Increased 
DoD  dependence  will  result  in  more  cyber  attacks  toward  the  poorly  protected  com¬ 
mercial  communication  satellites.  Caplan  [9]  also  states  that  Chinese  military  doc¬ 
trine  suggests  attacking  space-to-ground  communication  along  with  the  associated 
satellite  ground  stations  during  a  conflict.  China  is  also  actively  involving  several  of 
their  electronic  companies  in  intelligence  gathering  and  cyber  communications.  [9] 

A  specific  example  of  a  satellite  cyber  attack  occurred  in  2011  when  two  environ¬ 
mental  imaging  satellites  were  hacked.  The  hacker  was  able  to  access  the  satellite 
control  system  which  could  have  destroyed  the  satellite.  Fortunately  the  hacker  did 
not  issue  any  commands  leaving  the  satellite  unharmed.  A  statement  from  the  pres¬ 
ident  of  Intelsat  General  Corps  states,  “In  2011  alone,  IntelsatONE,  the  terrestrial 
network  that  links  customers  to  Intelsat’s  geosynchronous  communications  satellite, 
identified  about  300,000  denial-of-service  attacks”  [9].  Furthermore,  the  loss  of  the 
stealth  drone  in  2011  also  reveals  the  vulnerability  of  GPS  to  cyber  attacks.  Cyber 
threats  pose  a  serious  threat  to  space  systems  and  are  increasingly  prevalent.  [9] 
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Kinetic. 


Satellites  are  also  highly  vulnerable  to  kinetic  attacks.  In  2007,  China  destroyed  a 
weather  satellite  with  an  anti-satellite  missile  in  an  effort  to  shake  U.S.  dominance  in 
space  [1].  The  weather  satellite  was  in  LEO,  however,  a  Japanese  intelligence  official 
reports  that  China  is  hoping  to  target  GPS  satellites  in  future  conflicts  [1].  Lieutenant 
General  Michael  Hamel,  former  director  of  SMC,  states  that,  “[a]  ‘nightmare’  scenario 
is  for  multiple  satellites  used  by  the  Pentagon  to  ‘blink  off’,  indicating  a  hostile- 
possibly  kinetic  anti-satellite-campaign  against  the  U.S.  space  assets”  [8]. 

In  response  to  the  Chinese  ASAT  mission,  Senator  Jon  Kyi  recommends  several 
actions  which  should  be  taken  to  minimize  the  threat  [2],  Specifically,  Senator  Kyi 
states  that  efforts  should  be  focused  on  replacing  lost  satellites  quickly  [2],  Another 
option  is  to  quickly  identify  the  ASAT  threat  and  maneuver  the  targeted  satellite  to 
avoid  the  ASAT  [8].  However,  Colonel  Shawn  Barnes,  chief  of  AFSPC  Space  Superior¬ 
ity  Division,  states,  “[tjoday,  we  could  ascertain  that  we  were  under  attack-especially 
from  a  direct-ascent  [ASAT],  but  we  do  not  have  the  tools  to  rapidly  assemble  all  the 
evidence,  and  disseminate  it  in  a  way  that  enables  collaborative  decision  making”  [8]. 
Although  there  are  theoretical  options  available  to  counter  ASATs,  further  research 
is  needed  before  there  is  a  concrete  solution. 

2.4  PNT  Satellite  Systems 

Definition. 

Position,  navigation,  and  timing  satellites  are  used  in  both  military  and  civilian 
applications.  The  most  familiar  PNT  satellite  constellation  is  the  Global  Positioning 
System  which  is  owned  and  operated  by  the  LInited  States  Air  Force  [11].  GPS  is 
not  the  only  PNT  satellite  constellation  orbiting  Earth.  Several  other  countries  and 
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conglomerates  also  have  their  own  constellations.  The  European  Union  is  construct¬ 
ing  a  30  satellite  constellation  called  Galileo  for  enhanced  civilian  GPS  coverage  in 
conjunction  with  GPS  [33].  The  Russians  are  using  a  system  called  Glonass  and  the 
Chinese  are  using  their  system  called  Beidou.  Each  PNT  constellation  is  similar  in 
theory,  but  varied  in  application. 

The  concept  of  GPS  originated  with  NAVSTAR  developed  by  U.S.  government  in 
the  1970’s  [27].  GPS  has  continually  developed  and  today  we  are  launching  the  GPS 
Block  11F.  There  is  also  currently  a  contract  with  Lockheed  Martin  to  produce  GPS 
Ill  as  the  next  revolution  in  GPS  satellites  [24],  Impressively,  GPS  has  continued  to 
provide  uninterrupted  service  since  its  release  to  the  public  in  1995  [33].  The  military 
uses  GPS  for  navigation  and  guiding  precision  weapons,  “which  are  key  strategic 
weapons  for  the  United  States”  [1]  [27].  Civilians  also  use  GPS  for  a  multitude  of 
tasks  to  include  search  and  rescue,  land  surveying,  and  aircraft  collision  avoidance 
systems  just  to  name  a  few  [27]. 

GPS  satellites  nominally  operate  by  transmitting  a  ranging  signal  which  is  col¬ 
lected  at  a  GPS  receiver  [27].  Since  the  speed  of  the  GPS  signal  is  a  known  constant, 
the  GPS  receiver  can  take  the  difference  between  the  signal  transmit  time  and  received 
time  to  calculate  the  distance  from  the  satellite  to  the  receiver  [24],  Through  the  use 
of  atomic  clocks,  GPS  satellites  are  able  to  coordinate  their  time  stamps  associated 
with  their  ranging  signal,  which  is  essential  in  reducing  position  error  [27] .  Collecting 
distances  from  multiple  satellites  allows  the  GPS  receiver  to  locate  itself  based  upon 
the  only  possible  intersection  of  the  ranging  signal  from  the  satellites  [24] . 

Capabilities. 

In  order  to  ensure  the  highest  global  precision  for  military  efforts,  the  GPS  con¬ 
stellation  must  provide  a  minimum  of  24  operational  satellites  [11].  The  constellation 
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has  six  nominally  circular  planes  inclined  at  55  degrees  each  with  four  operational 
GPS  satellites  per  plane.  Each  GPS  satellite  has  a  period  of  12  hours.  Addition¬ 
ally,  a  GPS  receiver  must  be  able  to  receive  GPS  data  from  a  minimum  of  four  of 
the  24  available  satellites  in  order  to  provide  three-dimensional  location  information. 
The  more  satellites  a  receiver  can  process  and  the  larger  the  angle  between  received 
satellites,  the  better  the  position  estimation.  [27] 

There  are  several  options  for  GPS  constellation  improvement  that  may  increase 
overall  performance.  One  option  is  to  add  six  satellites,  one  more  in  each  orbital  plane, 
in  order  to  increase  coverage.  The  new  30  GPS  constellation  will  show  improvements 
in  the  number  of  GPS  satellites  available  to  the  receiver.  Another  option  is  to  add  an 
equatorial  orbital  plane  that  will  aid  in  providing  coverage  to  the  mid-latitude  regions 
which  are  most  susceptible  to  having  a  fewer  number  of  visible  GPS  satellites.  An 
increase  in  coverage  also  creates  another  plane  which  adds  to  the  complexity  of  the 
constellation.  A  final  consideration  is  changing  the  orbital  period  of  additional  GPS 
satellites  to  either  a  six  hour  orbit  in  a  lower  altitude  or  a  24  hour  geosynchronous  orbit 
in  a  higher  altitude  [5].  All  of  the  considerations  could  improve  the  GPS  constellation, 
however,  the  gained  performance  must  be  weighted  against  the  increased  complexity 
and  cost.  [27] 

Despite  the  benefits  of  the  GPS  constellation,  it  is  not  without  programmatic 
delays  and  error.  The  Government  Accountability  Office  (GAO)  provides  a  study 
from  2009  on  the  setbacks  experienced  with  the  GPS  constellation  upgrade.  The  study 
identifies  the  effects  of  operating  under  a  diminished  GPS  constellation.  For  example, 
the  precision  guided  munitions  used  by  the  military  would  be  less  accurate  leading 
to  larger  or  multiple  ordinances  necessary  to  ensure  the  same  mission  success  [1]  [11]. 
Additionally,  in  the  civilian  sector,  commercial  aircraft  and  911  emergency  responders 
would  also  be  effected  due  to  decreased  accuracy  from  GPS  location  estimates  [11]. 
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Urban  Canyon  Environment. 


The  urban  canyon  environment  is  one  type  of  challenging  environment  which 
is  characterized  by  tall  buildings,  long  narrow  streets  with  a  minimum  number  of 
intersections,  tunnels,  and  elevated  railways,  all  of  which  can  negatively  impact  GPS 
effectiveness  [34],  Vicek  et  al.  [34]  states  that,  “[r] effected  signals  and  relatively  poor 
geometries  make  GPS  derived  position  fixes  less  accurate  than  those  made  in  a  more 
benign  environment”.  Japan  experiences  habitually  unstable  GPS  service  due  to  the 
effects  of  their  urban  canyon  environment  [29].  As  a  result,  they  are  launching  four 
new  satellites  to  augment  the  GPS  constellation  over  Japan  in  order  to  reduce  multi 
path  errors  and  increase  satellite  availability  [29].  The  first  quasi- zenith  satellite  is 
currently  supporting  Japan  and  has  shown  an  improvement  from  39.5%  to  60.9% 
availability  over  one  of  the  test  cities  [15].  Continuing  plans  indicate  the  launch  of 
two  more  quasi-zenith  satellites  along  with  a  geosynchronous  satellite  by  2018  [29]. 
The  challenges  posed  by  an  urban  canyon  environment  are  a  reality  that  cannot  be 
ignored  in  a  combat  scenario. 

2.5  Agent-Based  Modeling 

Definition. 

Agent-based  modeling  and  simulation  (ABMS)  is  used  to  model  complex  adaptive 
systems  (CAS)  potentially  made  up  of  many  different  types  of  agents.  North  &  Macal 
[25]  state  that,  “Agent-based  modeling  and  simulation  (ABMS)  is  founded  on  the 
notion  that  the  whole  of  many  systems  or  organizations  is  greater  than  the  simple 
sum  of  its  constituent  parts”.  The  smaller  constituent  parts  can  in  turn  generate 
larger  emergent  behaviors.  One  of  the  principal  strengths  of  ABMS  is  to  identify 
the  connections  between  low  and  high  level  behavior.  A  simple  analogy  for  low  level 


18 


actions  causing  emergent  behavior  is  the  “wave”  commonly  produced  at  sporting 
events.  North  &  Macal  [25]  state  that,  “Each  person  or  agent  makes  small,  simple 
movements,  but  the  group  as  a  whole  produces  complex  large-scale  results”.  By 
accurately  modeling  the  agents  and  CAS  of  an  ABMS,  emergent  behaviors  can  be 
identified  and  assessed.  [25] 

ABMS  is  used  in  this  thesis  to  analyze  a  highly  complex  weapon  of  mass  destruc¬ 
tion  (WMD)  removal  scenario.  The  model  is  complex  due  to  the  multitude  of  support 
assets  to  the  blue  and  red  forces  which  have  a  unique  impact  on  the  ability  of  the 
special  operations  force  (SOF)  to  complete  the  mission.  It  is  important  to  remember 
that  the  agent  based  model  for  this  thesis  is  heavily  dependent  upon  the  blue  and 
red  force  agents.  However,  the  focus  of  the  study  is  on  the  satellites  supporting  the 
agents  during  the  simulation.  As  a  result,  the  efforts  of  this  thesis  is  not  on  re-coding 
the  agent  behavior,  but  instead  changing  the  CAS  supporting  the  agents. 

SEAS. 

The  System  Effectiveness  Analysis  Simulation  (SEAS)  is  a  computer  simulation 
developed  for  military  utility  analysis  which  supports  acquisition  programs  and  sys¬ 
tem  development.  SEAS  is  most  often  used  for  scenario  focused  simulations  between 
opposing  forces.  The  simulation  software  revolves  around  ABMS  which  allows  entities 
to  react  based  upon  their  perception  of  the  environment  and  their  pre-programmed 
rule  structure.  Agent-based  modeling  allows  entities  to  exhibit  complex-adaptive  be¬ 
havior  which  emerges  from  the  guiding  rules  structure  and  entity  interaction.  SEAS 
also  provides  a  visual  display  of  the  simulation  which  is  useful  for  verification  and 
debugging  portions  of  the  analysis.  Furthermore,  SEAS  is  a  part  of  the  Air  Force 
Standard  Analysis  Toolkit  and  continues  to  be  used  for  studies  in  military  analysis 
by  a  variety  of  users.  [32] 
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The  scenario  of  interest  has  been  previously  developed  in  SEAS  for  the  analysis  of 
the  GPS  in  an  urban  combat  environment.  The  code  contains  a  large  array  of  both 
blue  and  red  forces  governed  by  their  unique  rule  structure.  Each  of  the  coded  rule 
structures  contributes  to  the  emergent  behavior  of  the  entire  system.  Interpreting 
the  model  is  aided  by  the  visual  display  of  the  agents  moving  in  the  region  of  inter¬ 
est.  Furthermore,  SEAS  collects  simulation  statistics  which  can  be  transformed  into 
resiliency  metrics.  As  a  result,  SEAS  is  an  adequate  simulation  software  to  assess 
satellite  resiliency  in  our  scenario. 

2.6  Summary 

Accurately  modeling  satellite  resiliency  is  a  difficult  task  which  begins  by  estab¬ 
lishing  a  definition  for  resiliency.  The  definition  selected  for  the  study  comes  from  Air 
Force  Space  Command  which  focuses  on  a  system’s  ability  to  continue  providing  re¬ 
quired  capabilities  during  periods  of  degradation.  There  have  been  several  studies  on 
satellite  resiliency  along  with  a  variety  of  proposed  metrics.  Ultimately,  the  metrics 
selected  in  this  study  depend  upon  the  definition  of  resiliency  and  the  capabilities  of 
the  modeling  software.  Resiliency  has  become  an  important  topic  in  constellation  de¬ 
sign  because  the  increasing  number  of  threats  to  satellite  systems.  Today  more  than 
ever,  satellites  are  susceptible  to  kinetic,  cyber,  and  electromagnetic  threats  from  a 
variety  of  enemies.  PNT  constellations  in  particular  provide  a  crucial  capability  that 
must  be  maintained  if  at  all  possible.  In  order  to  assess  the  resiliency  of  a  satellite 
constellation,  the  agent-based  SEAS  program  is  used  to  model  a  scenario  impacted  by 
GPS  performance.  The  next  chapter  on  Methodology  will  further  develop  the  process 
for  assessing  the  resiliency  of  the  GPS  constellation  under  degradation. 
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III.  METHODOLOGY 


3.1  Overview 

The  Methodology  focuses  on  several  areas  connected  to  the  selected  scenario,  re¬ 
siliency  metrics,  model  developments,  and  the  proposed  analysis  techniques.  The  first 
section  provides  an  overview  of  the  selected  scenario  along  with  a  presentation  of  the 
logical  process  flow  and  a  description  of  the  blue  and  red  force  structures.  The  origi¬ 
nal  model  description  concludes  with  the  model  assumptions  and  input  parameters. 
Next,  we  present  the  chosen  resiliency  metrics  along  with  a  description  of  the  model 
developments.  After  the  model  developments  we  discuss  the  proposed  analysis  tech¬ 
niques,  which  includes  the  traditional  nominal  and  degraded  model  statistical  test 
comparison.  The  final  analysis  technique  describes  a  designed  experiment  which  is 
intended  to  assess  the  key  components  of  each  resiliency  metric.  Each  of  the  sections 
in  the  Methodology  builds  a  framework  for  the  analysis  of  the  selected  scenario. 

3.2  Scenario 

Description. 

The  selected  scenario  focuses  on  the  performance  of  the  GPS  constellation  in  an 
urban  canyon  environment  [14].  Air  Force  SMC/XR  Military  Utility  Analysis  (MUA) 
squadron  first  used  the  scenario  to  study  the  capabilities  of  GPS  satellites  in  stressed 
environments  [14].  Furthermore,  SMC/XR  selected  SEAS  to  develop  an  urban  canyon 
environment  due  to  its  short  build  up  time,  internal  resource  availability,  and  the 
ability  to  use  previous  studies  [14]. 

The  model  simulates  an  agent  based  SOF  team  moving  through  a  Middle  Eastern 
city  searching  for  a  WMD.  The  scenario  begins  with  the  SOF  team  landing  in  the 
city  and  heading  directly  to  the  WMD.  After  the  WMD  has  been  secured,  the  SOF 
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team  navigates  through  the  city  to  an  evacuation  location.  Successful  navigation  is 
aided  by  the  GPS  receiver  embedded  with  the  SOF  team.  Major  degradations  to  the 
GPS  constellation  can  cause  the  SOF  team  to  lose  its  position  knowledge  which  can 
lead  to  enemy  engagements  and  an  extend  mission  duration.  When  the  SOF  team 
successfully  makes  it  to  the  evacuation  location  the  mission  is  considered  a  success. 
[14] 

The  simulation  is  governed  by  a  complex  logic  process  which  is  focused  around 
the  SOF  team  as  they  transit  to  the  evacuation  location.  The  SOF  team  selects 
which  evacuation  location  is  the  best  option  and  navigates  to  the  location.  In  the 
process,  it  is  highly  likely  that  the  SOF  team  encounters  red  forces  which  can  deter 
the  evacuation  of  the  WMD.  Once  the  SOF  team  reaches  the  evacuation  location, 
they  will  secure  their  position  and  fend  off  enemy  advances  until  they  can  be  airlifted 
to  safety.  The  logical  process  flow  is  summarized  in  Figure  1. 
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Figure  1.  Logical  Process  Flow  [14] 


Model  Parameters. 

The  urban  canyon  scenario  has  several  input  parameters  that  can  be  changed  to 
reflect  desired  design  points.  The  first  four  parameters  are  GPS  constellation  specific 
which  represent  the  accuracy,  availability,  and  timeliness  of  the  GPS  constellation 
along  with  number  of  satellites  removed  due  to  degradation.  GPS  accuracy  is  defined 
in  meters  and  represents  the  position  accuracy  given  to  the  GPS  receiver  utilized  by 
the  SOF  team.  The  smaller  the  value  of  GPS  accuracy  represents  a  higher  effective¬ 
ness  of  the  GPS  constellation.  GPS  availability  is  scaled  from  0  to  1  and  represents 
the  probability  that  the  GPS  constellation  is  able  to  provide  geolocation  information. 
A  lower  probability  of  availability  is  used  to  simulate  the  impact  of  the  urban  en¬ 
vironment  which  may  reduce  the  chances  of  contacting  the  GPS  satellite.  Another 
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GPS  parameter  is  timeliness  which  reflects  the  amount  of  time  required  for  the  GPS 
receiver  to  provide  geolocation  information  to  the  SOF  team.  Additionally,  the  num¬ 
ber  of  GPS  satellites  removed  is  another  parameter  used  to  impact  the  constellation. 
For  example,  one  of  the  design  points  might  require  five  of  the  original  ten  satellites 
in  view  to  be  removed  from  the  simulation  to  reflect  a  period  of  degradation.  The 
combination  of  the  four  GPS  parameters  allows  for  top  level  mission  control  of  the 
GPS  constellation.  It  is  important  to  note  that  the  simulation  does  not  use  a  detailed 
GPS  geolocation  algorithm  or  provide  a  highly  detailed  perspective  on  the  satellite 
constellation.  Instead,  the  program  focuses  on  using  the  higher  level  input  parameters 
to  reflect  the  capabilities  of  the  GPS  constellation.  Each  of  the  GPS  parameters  and 
their  range  of  possible  values  are  summarized  in  Table  1.  [14] 

Table  1.  GPS  Parameters 

Input  Parameters 

Accuracy  Availability  Timeliness  Number  of  Satellites  Removed 
5  m  0.95  5  min  0 

40  m  0.34  5  min  10 


Range 

Best 

Worst 


Several  other  input  parameters  for  the  scenario  include  the  number  of  Predator 
surveillance  aircraft,  the  involvement  of  the  police,  and  the  ability  to  use  visual  cues. 
The  number  of  Predator  surveillance  aircraft  help  identify  red  force  threats  in  the 
simulation  as  they  circle  the  area  of  interest.  The  involvement  of  police  pertains  to 
the  ability  of  red  police  to  engage  the  SOF  team  in  combat  instead  of  just  identifying 
their  presence.  The  ability  to  use  visual  cues  corresponds  to  the  SOF  team  and  their 
ability  to  navigate  aided  by  landmarks  instead  of  only  using  the  GPS  receiver.  For 
this  study  these  parameters  remain  constant  and  are  not  adjusted  between  scenarios. 
As  a  result,  the  number  of  Predators  surveillance  aircraft  will  be  kept  at  two,  the 


24 


police  do  not  directly  contribute  to  military  engagements,  and  the  SOF  team  is  able 
to  use  visual  cues. 

Force  Composition. 

SOF  Team. 

The  principal  component  of  the  blue  forces  is  the  50  soldier  SOF  team.  Several 
rules  of  engagement  govern  the  actions  of  the  agent  based  SOF  team.  The  first  rule 
states  that  the  SOF  team  does  not  engage  in  combat  unless  they  are  bred  upon  from 
the  red  military  forces.  If,  however,  the  SOF  team  is  engaged  by  red  military  forces, 
then  they  are  drawn  into  the  fight.  The  SOF  retreats  from  the  battle  if  they  are 
outnumbered  by  a  3:1  margin.  Furthermore,  the  SOF  team  does  not  interact  with 
the  red  civilians  and  attempts  to  evade  the  red  police  forces.  Each  of  the  rules  of 
engagement  is  processed  at  every  time  step  in  the  simulation  in  order  to  determine 
the  SOF  team  actions.  [17] 

Additionally,  the  SOF  team  movements  are  heavily  reliant  upon  their  confidence  in 
the  GPS  geolocation  estimation  which  is  dependent  upon  the  number  of  GPS  satellites 
available.  For  example,  once  fewer  than  six  GPS  satellites  are  available,  then  the  SOF 
team  decreases  their  movement  speed.  The  most  extreme  impact  occurs  when  there 
are  fewer  than  four  GPS  satellites.  More  accurate  movements  of  the  SOF  team  leads 
to  a  shorter  mission  duration  with  a  minimal  loss  of  life.  [17] 

Blue  Forces. 

The  SOF  team  is  primarily  aided  by  the  GPS  constellation.  The  modeled  con¬ 
stellation  is  composed  of  the  32  current  GPS  satellites  which  are  listed  based  upon 
their  different  design  blocks.  The  Tactical  Programming  Language  (TPL)  code  for 
the  GPS  satellites  is  shown  in  Figure  2. 
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Figure  2.  GPS  Satellites  in  TPL  Code 


Each  of  the  satellites  is  governed  by  two  line  element  (TLE)  data  in  the  gps- 
ops.txt  hie  attached  to  the  overall  SEAS  scenario  warhle.  The  TLE  hies  have  all  been 
updated  as  of  November  2014.  An  example  of  the  TLE  code  is  annotated  and  shown 
in  Figure  3. 
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Satellite  Type 

Satellite  Number 


GPS 


>RN 


Epoch  Year 


1 

K-P-J.V  D 

1H103A 

14302.48952317 

. 00000000 

00000-0 

10000-3  0  5548 

2 

20959 

(54.2798 

203.5952  0114413 

355.9101 

4.07021 

2.00567260175224 

Orbital  Parameters:  Inclination,  RAAN,  Eccentricity,  Arg.  of  Perigee,  Mean  Anomaly 


Figure  3.  GPS  TLE  Example 


Each  of  the  GPS  satellites  orbit  the  Earth  as  dictated  in  the  TLE  data  and  are 
included  in  the  target  list  for  the  GPS  Unit.  The  GPS  Unit  provides  input  to  the 
GPS  receiver  which  is  used  by  the  SOF  team.  The  SOF  team  is  most  aided  by  a  GPS 
constellation  that  has  a  low  value  for  accuracy,  high  probability  of  availability,  low 
timeliness,  and  a  high  number  of  visible  GPS  satellites. 

The  blue  forces  are  also  composed  of  several  additional  support  assets  which  pro¬ 
vide  airlift  and  surveillance.  Airlift  is  accomplished  through  the  CV-22  which  deliv¬ 
ers  the  SOF  team  in  the  city  and  recovers  the  SOF  team  from  the  evacuation  point. 
Surveillance  is  made  possible  through  the  interactions  of  several  different  platforms  in 
continual  communication.  The  Predator  and  Global  Hawk  units  help  identify  enemy 
red  targets  throughout  the  city  as  they  circle  over  the  area  of  interest.  Additionally, 
a  carrier  support  group  is  included  in  the  simulation  to  help  relay  the  information 
between  the  blue  forces.  The  combination  of  the  support  elements  are  not  specifically 
studied  in  this  analysis,  however,  it  is  important  to  understand  that  they  provide  the 
transportation  and  communication  structure  for  the  blue  forces. 

Red  Forces. 

The  red  force  structure  is  composed  of  several  generic  unit  categories  to  include 
military  forces,  police,  and  civilians.  Red  military  units  are  the  main  threat  to  the 
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SOF  team  and  engage  in  combat  whenever  possible.  The  red  military  is  composed 
of  six  trucks  with  11  soldiers  per  truck.  If  the  red  military  engages  the  SOF  team 
but  the  number  of  red  military  forces  do  not  exceed  three  times  the  number  of  blue 
forces,  then  the  red  forces  retreat.  The  red  forces  follow  the  blue  forces  until  red 
reinforcements  arrive.  The  red  military  is  also  programmed  to  act  autonomously  from 
the  red  police,  however,  the  red  military  reacts  to  SOF  team  locations  transmitted 
over  the  red  police  radio.  The  red  police  are  less  numerous  and  less  threatening  to 
the  SOF  team.  They  do  not  engage  the  SOF  team,  but  proceed  about  their  normal 
patrols.  The  police  act  regionally  around  their  five  different  police  stations  and  use 
motorcycles  and  station  wagons  as  fast  transportation.  If  the  red  police  encounter  the 
SOF  team,  they  broadcast  the  SOF  team  location  which  can  be  intercepted  by  the 
red  military.  Additionally,  the  police  can  be  informed  about  the  SOF  team  location 
from  red  civilians.  The  red  civilians  continue  to  grow  in  number  throughout  the 
simulation  and  move  randomly  throughout  the  city.  If  a  civilian  encounters  the  SOF 
team,  then  there  is  a  50  percent  chance  that  they  inform  the  police.  The  complex  red 
force  structure  is  heavily  dependent  upon  the  communication  of  SOF  team’s  current 
location.  If  the  civilians  and  police  can  communicate  effectively,  then  the  red  military 
will  be  more  successful  at  engaging  the  SOF  team.  [17] 

Model  Assumptions. 

The  urban  canyon  scenario  has  several  assumptions  built  into  the  model.  A  list 
of  the  model  assumptions  are  summarized  below  [17]: 

1.  GPS  accuracy  is  degraded  when  the  SOF  team  is  near  a  building. 

2.  An  agent’s  position  knowledge  is  impacted  by  input  parameters. 


3.  A  specific  geolocation  algorithm  is  not  necessary. 


4.  The  SOF  team  must  maneuver  through  the  streets  to  reach  the  evacuation 
location. 

5.  The  CV-22  is  not  impacted  by  GPS  degradation. 

The  first  critical  assumption  is  the  degradation  of  GPS  accuracy  as  the  SOF  team 
moves  closer  to  buildings.  One  example  provided  from  the  model  designers  is  that 
the  accuracy  of  GPS  can  decrease  from  2-m  to  30-m  when  the  SOF  team  moves  to 
within  a  certain  distance  to  the  buildings  [17].  Another  important  assumption  in 
the  model  is  that  the  SOF  team’s  position  knowledge  is  adjusted  by  three  model 
input  parameters,  GPS  accuracy,  availability,  and  timeliness  [17].  Furthermore,  the 
simulation  does  not  use  a  specific  geolocation  algorithm,  but  instead  relies  on  top-level 
characteristics  of  GPS  to  determine  position.  Each  of  the  assumptions  help  construct 
a  top-level  mission  model  as  opposed  to  a  more  detailed  tactical  engagement  level 
model. 

3.3  Model  Development 

Tactical  Programming  Language. 

Agents  within  SEAS  are  governed  by  logic  found  in  the  TPL  code.  In  the  sce¬ 
nario  simulation  code  there  is  a  section  for  “orders”  for  each  agent  where  the  TPL 
statements  are  located.  The  TPL  associate  logic  statements  with  each  agent  which 
enable  specific  behavior.  Conventional  logical  statements  are  available  in  the  TPL 
to  include,  arrays,  variable  manipulation,  condition  statements,  logic  loops,  and  a 
multitude  of  both  user  defined  and  pre-dehned  functions.  Minimal  changes  to  the 
TPL  code  are  implemented  in  the  scenario  of  interest  to  maintain  the  validity  of  the 
current  model.  Nevertheless,  understanding  the  syntax  and  proper  implementation 
of  different  TPL  statements  is  essential  to  ensuring  accurate  analysis.  [18] 
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Modifications. 


To  maintain  the  current  model  validity,  only  minor  changes  are  made  to  the  urban 
canyon  scenario.  Additionally,  the  analysis  does  not  require  any  additional  output 
variables  from  the  model  which  greatly  simplifies  the  model  development.  Instead, 
model  development  focuses  on  logic  structure  changes  along  with  the  changes  in 
the  number  of  available  GPS  satellites.  Each  of  the  model  developments  are  used  in 
combination  with  input  parameter  changes  to  create  the  degraded  scenarios  described 
in  the  Analysis  Approach  section. 

One  modification  aims  to  remove  available  satellites  from  the  model.  Removing 
satellites  is  key  to  assessing  overall  constellation  resiliency  and  can  be  accomplished 
by  selectively  commenting  out  satellites.  Any  number  of  satellites  can  be  randomly 
or  purposely  selected  within  the  model  for  omission.  One  example  is  the  removal  of 
the  most  recently  deployed  GPS  Block  IIF  series  satellites  of  which  there  are  seven 
included  in  the  original  simulation.  To  remove  satellites  from  consideration,  they 
must  be  commented  out  from  both  the  group  definition  of  the  GPS  Sats.inc  file  along 
with  the  satellite  force  definition  in  the  pnt.war  file.  Removing  satellites  can  be 
representative  of  many  real  world  threats  to  include,  kinetic  strikes,  cyber  attacks, 
environmental  weather  variations,  and  system  failure. 

The  second  modification  is  an  addition  to  the  GPS  Unit  logic  structure.  In  the 
original  model,  the  SOF  team  would  receive  a  decrease  in  overall  travel  speed  if  less 
than  six  GPS  satellites  were  available  on  the  target  list.  To  expand  the  original  model 
and  connect  the  GPS  Unit  more  directly  to  the  scenario,  the  logic  statements  also 
include  changes  to  the  input  parameters  of  GPS  accuracy  and  availability.  Addition¬ 
ally,  the  logic  statements  are  expanded  to  provide  changes  to  the  input  parameters 
for  every  number  of  available  GPS  satellites  under  ten.  The  input  parameters  are 
changed  through  a  increasing  linear  scale  for  GPS  availability  and  a  decreasing  expo- 
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nential  scale  for  GPS  accuracy  as  the  number  of  satellites  on  the  target  list  increases. 
A  view  of  the  proposed  factors  in  the  logic  development  is  shown  in  Table  2. 


Table  2.  GPS  Unit  Parameter  Factors  per  Number  of  GPS  Satellites 


Multiplication  Factor 

Number  of  Visible  GPS  Satellites 

Accuracy 

Availability 

Speed 

<  4 

4 

0.4 

0.5 

5 

3.17 

0.5 

0.75 

6 

2.52 

0.6 

0.9 

7 

2 

0.7 

0.9 

8 

1.59 

0.8 

0.9 

9 

1.26 

.9 

0.9 

10 

1 

1 

1 

The  logic  factors  are  translated  into  specific  values  for  accuracy  and  availability 
based  on  either  nominal  or  degraded  operations.  The  realized  values  are  summarized 
in  Figures  4  and  5. 
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GPS  Accuracy 


•Nominal 

•Degraded 


Figure  4.  Translated  GPS  Accuracy 


Figure  5.  Translated  GPS  Availability 
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3.4  Selected  Metrics 


The  analysis  in  this  study  focuses  on  answering  the  fundamental  question  of  GPS 
constellation  resiliency.  As  a  result  of  the  challenges  noted  in  the  Literature  Review, 
the  metrics  needed  to  capture  GPS  resiliency  must  be  insightful  and  cleverly  devel¬ 
oped.  Furthermore,  the  metrics  must  be  accessible  within  the  current  scenario  and 
cannot  require  drastic  changes  in  the  SEAS  TPL  code.  Limiting  the  changes  in  the 
code  and  utilizing  already  available  model  outputs  allow  the  model  to  maintain  its 
validity.  In  order  to  derive  the  appropriate  metrics,  a  series  of  top  level  mission  pri¬ 
orities  are  established.  Each  of  the  selected  metrics  connect  back  to  one  of  the  four 
campaign  priorities.  The  list  of  campaign  priorities  are  shown  below: 

1.  WMD  recovery  at  all  costs; 

2.  Minimize  blue  force  casualties; 

3.  Minimize  mission  duration;  and 

4.  GPS  is  functioning  as  intended. 

Probability  that  more  than  50%  of  SOF  Team  Survives  (P5 0). 

P50  is  derived  from  the  number  of  replications  where  fewer  than  50%  casualties 
are  sustained  by  the  SOF  team.  The  metric  connects  back  to  the  first  and  second 
campaign  level  priorities  which  reflect  the  proportion  of  time  that  the  WMD  has  been 
successfully  removed  with  fewer  than  25  SOF  team  casualties.  Instead  of  stopping  the 
simulation  once  the  SOF  team  casualties  exceed  25,  we  chose  to  allow  the  simulation 
to  continue  running  which  represents  a  need  to  remove  the  WMD  at  all  costs.  As  a 
result,  the  mission  is  classified  through  the  output  data  by  observing  if  there  are  more 
than  25  casualties  in  the  replication.  If  the  metric  varies  greatly  across  scenarios  then, 
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it  is  possible  to  argue  that  degraded  GPS  is  hindering  the  mission  and  the  constellation 
is  not  exhibiting  resiliency. 

Mission  Duration. 

Mission  duration  assesses  the  impact  of  being  lost  as  a  result  of  GPS  degradation. 
The  metrics  connects  directly  to  the  third  campaign  level  priority  which  aims  to 
reduce  the  total  duration  of  the  mission.  An  extend  mission  duration  can  occur 
if  the  GPS  constellation  is  degraded  by  having  a  fewer  number  of  satellites  which 
decreases  accuracy  and  availability  and  increases  the  probability  of  being  lost.  Any 
additional  time  lost  is  a  sign  of  failure  of  the  GPS  constellation  to  provide  the  required 
capabilities  indicating  a  lack  of  resiliency. 

Number  of  Blue  Casualties. 

The  number  of  blue  casualties  connects  to  the  second  campaign  level  priority 
and  provides  insight  into  the  SOF  team  losses.  The  metric  is  designed  to  assess  the 
number  of  casualties  as  a  result  of  degraded  GPS  as  opposed  to  poor  battle  tactics. 
Intuitively,  the  number  of  casualties  should  increase  with  an  increase  in  the  number  of 
engagements.  Reducing  the  number  of  engagements  is  heavily  dependent  upon  correct 
position  data  from  the  GPS  constellation.  As  a  result,  significant  differences  in  the 
amount  of  blue  casualties  across  scenarios  reflects  a  weakness  and  lack  of  resiliency 
of  the  GPS  constellation. 

Number  of  Engagements. 

The  number  of  engagements  is  intertwined  with  the  number  of  blue  casualties 
and  is  therefore  also  connected  to  the  second  campaign  level  priority.  Reducing  the 
number  of  engagements  leads  to  a  decreased  loss  of  life  along  with  a  decreased  mission 
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duration.  The  best  way  to  avoid  engagements  is  to  move  precisely  through  the  city 
to  the  evacuation  point.  The  number  of  engagements  is  included  along  with  the 
number  of  casualties  to  help  decipher  if  GPS  is  having  an  impact  which  may  be 
disguised  by  a  resilient  SOF  team  which  does  not  suffer  a  large  amount  of  casualties 
per  engagement.  As  a  result,  if  the  number  of  engagements  significantly  increases 
with  GPS  degradation,  then  it  is  possible  to  conclude  that  the  GPS  constellation  is 
not  providing  the  required  capability  and  is  therefore  not  exhibiting  resiliency. 

Blue  Casualties  per  Number  of  Engagements. 

As  noted  earlier  in  the  Literature  Review,  a  ratio  metric  can  provide  more  infor¬ 
mation  beyond  just  the  output  from  a  single  variable  [12].  Nevertheless,  the  ratio 
between  the  number  of  casualties  and  the  engagements  did  not  yield  any  interesting 
results  or  insight  throughout  the  analysis.  As  a  result,  the  specific  ratio  metric  is  not 
considered  for  the  study. 

Proportion  of  Time  Lost  to  Mission  Duration. 

The  proportion  between  the  amount  of  time  the  SOF  team  is  lost  and  the  total 
mission  duration  connects  directly  to  the  third  campaign  level  priority  which  aims  to 
reduce  the  total  mission  duration.  When  the  SOF  team  losses  its  position  knowledge 
due  to  a  degraded  GPS  constellation  and  is  lost,  the  intuitive  result  is  an  increased 
mission  duration.  It  can  be  difficult  to  interpret  the  specific  values  of  the  metric, 
however,  any  increase  in  the  metric  indicates  an  increased  amount  of  time  being 
wasted.  A  statistical  difference  comparison  between  the  scenarios  provides  insight 
into  GPS  constellation  resiliency. 
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Percentage  of  Time  that  the  Number  of  GPS  Satellites  is  less  than  4. 


The  average  percentage  of  time  that  less  than  four  GPS  satellites  are  in  view 
reflects  the  functionality  of  the  constellation.  GPS  is  specifically  designed  to  provide 
accurate  geolocation  data  to  a  receiver  with  a  minimum  of  four  GPS  satellites  [27].  If 
the  number  of  GPS  satellites  in  view  drops  below  four,  then  it  is  reasonable  to  argue 
that  the  SOF  team  movements  are  significantly  impacted  and  the  successful  removal 
of  the  WMD  is  diminished.  The  metric  specifically  relates  back  to  the  fourth  priority 
which  is  concerned  with  the  functionality  of  GPS.  If  the  GPS  constellation  is  not 
functioning  as  intended,  then  from  a  top  level  mission  perspective,  the  substantial 
monetary  investment  in  GPS  is  not  providing  the  expected  return  in  capability. 

Percentage  of  Time  that  the  Number  of  GPS  Satellites  is  greater  than 

6. 

A  similar  metric  is  connected  to  the  functionality  and  intended  operation  of  the 
GPS  constellation.  While  the  previous  metric  aims  to  collect  data  on  the  minimal 
number  of  satellites  required  for  geolocation,  the  GPS  constellation  is  ideally  created 
to  provide  highly  accurate  geolocation  achievable  from  a  large  number  of  satellites. 
Collecting  statistics  on  having  more  than  six  GPS  satellites  provides  information  on 
the  portion  of  the  mission  which  is  receiving  high  quality  GPS  information.  As  the 
model  is  degraded,  it  may  not  be  possible  for  the  GPS  constellation  to  retain  more 
than  six  satellites  in  view  which  indicates  a  loss  of  resiliency  to  provide  the  high 
resolution  data  to  the  SOF  team. 
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3.5  Analysis  Approach 


Scenario  Comparison. 

The  first  method  of  analysis  compares  the  set  of  selected  metrics  across  modified 
scenarios  using  statistical  difference  tests.  Each  scenario  is  defined  by  either  a  nominal 
or  degraded  starting  operational  condition  which  can  be  further  degraded  by  removing 
a  number  of  the  satellites  attained  from  the  initial  target  list.  The  objective  is  to 
determine  if  there  is  a  significant  difference  between  the  metrics  from  one  scenario 
to  another.  We  use  the  Paired  T  test  for  any  continuous  or  integer  metrics.  For  any 
percentage  or  proportion  metric  we  perform  a  classical  difference  of  proportions  test 
in  place  of  a  Paired  T. 

Nominal  Operations. 

In  an  effort  to  represent  current  scenario  performance,  the  nominal  operations  are 
defined  by  expected  GPS  constellation  input  parameters.  The  two  most  important 
parameters  are  the  GPS  accuracy  and  availability  which  are  set  at  5  meters  and  95% 
respectfully.  It  is  reasonable  to  assume  that  the  GPS  constellation  is  not  perfect,  but  it 
is  however  highly  accurate  and  readily  available.  GPS  timeliness  is  kept  at  five  seconds 
which  was  the  fastest  value  used  by  the  previous  model  developers.  Additionally,  the 
model  includes  two  Predators,  the  police  do  not  engage  in  combat,  and  the  SOF  team 
is  able  to  use  landmarks  as  mentioned  previously. 

Degraded  Operations. 

The  degraded  operations  reflect  a  minor  loss  of  GPS  capability.  The  only  changes 
made  to  the  nominal  operations  are  the  reduction  of  GPS  accuracy  and  availability. 
GPS  accuracy  is  increased  to  10  meters  and  the  availability  is  reduced  to  85%.  The 
degraded  scenario  represents  the  impact  of  the  urban  canyon  environment  where  the 
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geolocation  estimates  can  be  impacted  along  with  the  connection  to  the  GPS  satellites. 
The  remaining  input  variables  remain  the  same  as  those  in  the  nominal  operations. 

Satellite  Removal. 

In  order  to  simulate  the  loss  of  satellites  due  to  the  array  of  threats  mentioned  in 
the  Literature  Review,  satellites  are  purposefully  eliminated  from  the  program  code. 
The  satellites  selected  for  removal  are  attained  from  the  initial  target  list  of  the  GPS 
Unit  and  remain  omitted  for  the  duration  of  the  simulation.  Using  the  target  list 
enables  the  removal  of  the  satellites  specifically  in  view  of  the  area  of  operations, 
instead  of  removing  a  satellite  at  random  which  may  not  be  important.  This  also 
implies  a  red  force  capability  to  identify  and  rapidly  target  specific  satellites.  The 
number  of  satellites  removed  ranges  from  zero  to  ten  of  the  initial  satellites  and  is 
used  in  combination  with  either  the  nominal  or  degraded  operations  scenarios. 

Designed  Experiment. 

A  full  factorial  designed  experiment  is  performed  which  provides  insight  into  the 
most  important  variables  impacting  the  selected  responses.  The  responses  of  interest 
are  the  selected  metrics  to  assess  resiliency.  Additionally,  the  designed  experiment  is 
composed  of  two  factors.  The  first  factor  is  the  use  of  either  the  nominal  or  degraded 
operations,  while  the  second  factor  is  structured  on  the  number  of  satellites  removed. 
Using  the  results  from  the  designed  experiment,  it  is  possible  to  determine  which 
factor  most  impacts  the  overall  resiliency. 

3.6  Summary 

The  WMD  removal  scenario  provided  by  SMC  is  highly  complex  yet  adaptable  for 
additional  analysis  on  the  GPS  constellation  resiliency  in  an  urban  canyon  environ- 
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ment.  The  simulation  is  composed  of  opposing  blue  and  red  forces  which  are  assisted 
by  a  range  of  support  assets.  The  simulation  revolves  around  the  actions  of  the  SOF 
team  which  maneuvers  through  the  city  to  evacuate  the  WMD.  During  the  maneuver 
period,  the  SOF  team  is  continually  confronted  by  red  military  forces  in  their  attempt 
to  eliminate  the  SOF  team.  A  principle  support  structure  for  the  SOF  team  is  the 
GPS  constellation  providing  navigation  knowledge.  The  agent  based  actors  in  the 
model  are  limited  by  several  assumptions  to  include  GPS  degradation  when  the  SOF 
team  is  near  to  buildings.  To  further  augment  the  degradation,  several  changes  are 
made  to  the  TPL  code  in  SEAS  model.  The  first  change  is  to  selectively  comment 
out  specific  satellites  from  the  model.  The  second  change  is  to  the  GPS  Unit  logic 
which  decreases  the  GPS  accuracy  and  availability  characteristics  as  the  number  of 
satellites  in  view  decrease.  To  assess  the  resiliency  of  the  GPS  constellation,  a  series 
of  metrics  are  selected  to  support  the  list  of  campaign  level  priorities.  The  metrics  are 
then  assessed  across  scenarios  to  determine  if  there  is  a  statistical  difference  which 
provides  insight  into  resiliency.  Furthermore,  a  designed  experiment  helps  provide 
additional  insight  into  the  most  important  factors  behind  a  specific  metric.  Chapter 
4  presents  the  analysis  derived  from  the  Methodology  along  with  insights  gained  into 
GPS  constellation  resiliency. 
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IV.  ANALYSIS 


4.1  Overview 

Chapter  4  presents  the  analysis  performed  as  described  in  the  Methodology  and 
the  results  attained.  First,  the  appropriate  sample  size  is  determined  for  each  scenario 
along  with  the  required  number  of  model  runs.  Next,  the  initial  investigation  into  the 
data  is  performed  through  a  series  of  Paired  T  or  difference  of  proportions  tests  and 
confidence  intervals  to  determine  if  there  is  a  statistically  significant  difference  in  the 
metrics  across  scenarios.  Those  metrics  which  appear  to  be  important  are  identified 
and  selected  for  further  analysis.  The  analysis  section  also  describes  the  process  of 
attaining  all  of  the  production  replications  required  for  the  designed  experiment.  Once 
all  of  the  data  is  collected,  a  graphical  analysis  of  the  data  is  presented.  Furthermore, 
a  series  of  linear  regression  models  are  generated  to  determine  which  factors  are  most 
influential  for  a  specific  metric.  The  Analysis  chapter  concludes  with  a  brief  summary 
of  the  results  which  is  extended  upon  in  Chapter  5. 

4.2  Sample  Size  and  Model  Runs 

The  number  of  replications  for  each  design  point  is  driven  by  the  need  to  provide 
accurate  analysis  balanced  with  the  resources  required  to  complete  one  replication. 
Fortunately,  each  replication  of  the  simulation  only  requires  several  minutes  on  a 
personal  laptop.  However,  it  is  not  advantageous  to  perform  as  many  replications 
as  possible  as  the  information  gained  from  the  additional  replications  experiences  a 
diminishing  margin  of  return.  Instead,  the  sample  size  is  determined  to  provide  a 
reasonable  standard  deviation  relative  to  the  mean  value  for  all  metrics.  We  looked 
closely  at  results  for  20,  25,  and  30  replications  and  selected  25  replications  as  a  good 
balance  across  all  metrics.  For  the  remainder  of  the  study  each  design  point  contains 
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25  replications. 

Model  parameters  for  the  normal  and  degraded  operations  can  be  varied  within 
a  single  SEAS  warfile  with  the  use  of  the  run  matrix  logic  included  in  the  original 
model.  By  choosing  to  remove  satellites  by  commenting  out  appropriate  portions  of 
the  TPL  code  within  a  SEAS  warble,  each  change  in  number  of  satellites  removed 
requires  a  modibed  warble.  Therefore,  the  study  runs  are  broken  into  50  replications 
as  further  discussed  in  this  chapter. 

4.3  Initial  Analysis 

To  gain  an  initial  perspective  on  the  data,  an  initial  analysis  is  composed  of  a 
variety  of  scenarios  at  the  extreme  and  moderate  levels  of  each  factor.  The  metrics 
for  each  scenario  are  compared  to  determine  if  there  is  a  statistically  signibcant  diber- 
ence.  The  presence  of  a  statistical  cliberence  indicates  a  metric  which  is  impacted  by 
the  factor  level  changes  to  the  simulation.  While  initial  insight  is  possible  through  all 
metrics,  the  metrics  which  do  exhibit  a  statistical  cliberence  are  considered  for  further 
analysis.  Additionally,  the  sensitivity  of  a  metric  to  the  initial  design  points  provides 
information  on  the  range  of  the  factor  levels  used  in  the  designed  experiment.  An¬ 
other  beneht  of  the  initial  analysis  is  the  generation  of  the  conhdence  intervals  which 
display  the  expected  range  of  cliberence  for  each  metric.  The  magnitude  and  range 
of  the  conhdence  interval  provides  information  into  the  extent  of  the  cliberence.  Fur¬ 
thermore,  observing  patterns  and  anomalies  in  the  results  leads  directly  to  inferences 
on  the  resiliency  of  the  GPS  constellation. 

Selected  Scenarios. 

The  six  selected  initial  scenarios  represent  the  center  and  end  design  points  be¬ 
tween  of  the  number  of  satellites  removed  and  the  type  of  operations.  By  examining 
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the  extreme  scenarios  in  conjunction  with  moderate  scenarios  allows  for  greater  ini¬ 
tial  insight  over  the  range  of  the  study.  The  scenarios  are  performed  in  pairs  between 
nominal  and  degraded  operations  where  the  number  of  satellites  removed  is  held  con¬ 
stant.  As  a  result,  each  run  matrix  for  this  initial  simulation  contains  50  replications 
of  which  the  first  25  are  allocated  to  the  nominal  operations  and  the  remaining  25 
replications  to  the  degraded  operations.  Table  3  displays  the  initial  scenarios. 


Table  3.  Initial  Analysis  Scenarios 


Scenario 

Availability 

Accuracy  (m) 

Number  of  Satellites  Removed 

Reps 

1 

0.95 

5 

0 

25 

2 

0.85 

10 

0 

25 

3 

0.95 

5 

5  (9,  1,  31,  17,  23) 

25 

4 

0.85 

10 

5  (9,  1,  31,  17,  23) 

25 

5 

0.95 

5 

10  (9,  1,  31,  17,  23,  20,  11,  8,  4,  32) 

25 

6 

0.85 

10 

10  (9,  1,  31,  17,  23,  20,  11,  8,  4,  32) 

25 

Metric  Averages. 

The  raw  output  data  from  the  scenarios  shown  in  Table  3  provide  information  on 
the  mean  values  of  the  metrics  in  the  analysis.  A  summary  of  the  mean  values  are 
presented  in  Tables  4  and  5  for  the  nominal  and  degraded  operations  respectfully. 


42 


Table  4.  Metric  Averages  for  Nominal  Operations 


Metric 

Number  of  Satellites  Removed 

0  5  10 

Pm 

68.00% 

44.00% 

8.00% 

Duration  (min) 

73.00 

80.52 

120.31 

Casualties 

16.76 

19.76 

33.44 

Engagements 

4.44 

6.64 

10.24 

%  Time  Lost 

0.37% 

1.12% 

2.35% 

%  <  4  GPS 

0.25% 

1.70% 

45.71% 

%  >  6  GPS 

63.61% 

54.35% 

7.46% 

Table  5.  Metrics  Averages  for  Degraded  Operations 


Metric 

Number  of  Satellites  Removed 

0  5  10 

Pm 

40.00% 

24.00% 

12.00% 

Duration  (min) 

84.13 

100.21 

139.35 

Casualties 

23.68 

28.44 

33.00 

Engagements 

7.12 

10.72 

11.20 

%  Time  Lost 

1.31% 

1.97% 

3.12% 

%  <  4  GPS 

0.67% 

3.04% 

40.68% 

%  >  6  GPS 

66.85% 

59.01% 

15.76% 

Both  Tables  4  and  5  support  intuitive  trends  across  scenarios.  For  example, 
while  holding  the  operations  constant  as  GPS  satellites  are  removed,  the  scenarios 
report  on  average  a  lower  value  for  P50,  longer  duration,  higher  number  of  casualties, 
higher  number  of  engagements,  an  increased  percentage  of  time  lost,  and  fewer  satel¬ 
lites  available  to  the  GPS  unit.  Holding  the  number  of  satellites  removed  constant 
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and  looking  across  operations  generally  shows  that  the  degraded  operations  perform 
poorly  when  compared  to  the  nominal  operations.  There  are  a  few  exceptions  where 
the  degraded  operations  performs  better  than  the  nominal  operations;  however,  it 
is  important  to  remember  that  these  are  only  mean  value  comparisons  and  do  not 
contain  an  assessment  of  a  statistical  difference.  Nevertheless,  Tables  4  and  5  provide 
an  intuitive  first  perspective  to  the  metrics. 

Paired  T  and  Difference  of  Proportions  Comparisons. 

The  scenario  comparisons  gain  statistical  rigor  through  the  use  of  the  Paired  T  test 
on  the  duration,  casualties,  and  engagement  metrics  while  the  difference  of  propor¬ 
tions  confidence  interval  is  used  on  the  remaining  metrics  which  are  all  percentages. 
There  are  several  ways  to  compare  point  estimators  for  the  mean;  however,  the  Paired 
T  test  is  the  least  restrictive  in  terms  of  assumptions.  One  condition  for  the  Paired  T 
test  is  that  the  differences  between  the  paired  replications  should  be  approximately 
normally  distributed  [35].  Fortunately,  the  Paired  T  test  is  rather  robust  to  non¬ 
normal  data.  As  a  result,  with  a  sample  size  of  25  replications,  we  assume  that  the 
difference  data  is  approximately  normally  distributed  and  passes  the  first  condition. 
Typically,  the  Paired  T  test  is  required  for  dependent  data  which  can  be  achieved  by 
using  synchronized  random  numbers.  In  this  study,  we  do  not  attempt  to  synchro¬ 
nize  the  random  numbers  in  the  simulation  and  as  a  result  we  can  conclude  that  the 
replications  are  independent.  Therefore,  the  Paired  T  test  is  an  acceptable  method 
for  comparing  the  numerical  metrics. 

The  compilation  of  the  Paired  T  test  and  confidence  interval  provide  insight  into 
the  difference  between  the  metrics  in  each  scenario.  The  p-value  from  the  Paired  T 
test  provides  information  on  the  extent  of  the  null  hypothesis  conclusion  to  either 
reject  or  fail  to  reject  a  difference  between  the  metrics  in  the  scenario  comparison.  In 
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addition,  the  confidence  interval  displays  the  point  estimator  of  the  difference  along 
with  a  range  which  contains  95%  of  the  observed  differences.  For  cases  where  we  reject 
the  null  hypothesis  at  a  =  0.05  level  of  significance,  the  95%  confidence  interval  does 
not  contain  zero.  Both  of  these  results  indicate  a  statistically  significant  difference 
between  the  scenarios. 

The  proportion  metrics  are  not  compared  with  the  Paired  T  test,  but  instead 
through  the  difference  in  proportions  confidence  interval.  The  data  contained  in  each 
of  the  replications  for  P50  are  distributed  binomially  as  either  a  success  or  failure.  As 
a  result,  the  differences  in  the  Paired  T  test  are  not  normally  distributed  or  insightful. 
Additionally,  the  remaining  proportion  metrics  cannot  be  assessed  through  the  Paired 
T  test  due  to  inaccuracies  in  the  standard  error  calculations.  Instead,  the  appropriate 
comparison  method  is  to  determine  if  the  difference  in  proportions  confidence  interval 
contains  zero.  The  proportion  confidence  interval  is  shown  in  Equation  1  [19]  where 
Pi  and  p2  represent  the  proportion  correct  from  the  replications  of  each  scenario. 


Pi-p2=Pi-p2±  A 


0.025 


\ 


Pi(i-Pi)  p2(i-p2) 

+ 


n  1 


n2 


(1) 


Two  key  differences  between  the  proportion  and  Paired  T  test  confidence  interval 
are  the  standard  error  and  distribution  statistic.  The  standard  error  for  a  propor¬ 
tion  requires  a  pooled  variance  estimation  from  both  populations.  Each  individual 
variance  is  also  calculated  by  taking  the  product  of  the  proportion  correct  and  the 
proportion  of  failure  and  the  total  number  of  replications.  Furthermore,  the  difference 
of  proportions  interval  uses  a  z  statistic  in  place  of  the  t  statistic  used  for  the  Paired 
T. 
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Initial  Comparison. 


Several  important  trends  are  identified  from  the  initial  analysis  which  provide 
insight  into  satellite  resiliency  and  direct  the  remainder  of  the  analysis.  A  compiled 
table  of  the  p-value  results  for  each  of  the  metric  comparisons  across  the  different 
scenarios  are  shown  Tables  6,  8,  and  10.  Note  that  p-values  are  provided  for  the 
Paired  T  test  and  either  a  “Reject”  (p-value  <  0.05)  or  fail  to  reject  “FTR”  for 
the  difference  of  proportions.  Additionally,  the  confidence  interval  for  each  metric 
comparison  is  summarized  with  the  mean  and  half  width  shown  in  Tables  7,  9, 

and  11.  Bolded  cells  indicate  that  there  is  a  statistically  significant  difference  for  the 
metric  in  the  given  scenario  comparison. 
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Table  6.  Nominal  vs  Degraded  P- Value  Comparison 


Metric 

Nom-0  vs  Degrad-0 

Scenario  Comparison 

Nom-5  vs  Degrad-5  Nom-10  vs  Degrad- 10 

Pm 

Reject 

FTR 

FTR 

Duration  (min) 

0.01 

0.00 

0.01 

Casualties 

0.13 

0.02 

0.82 

Engagements 

0.03 

0.01 

0.13 

%  Time  Lost 

FTR 

FTR 

FTR 

%  <  4  GPS 

FTR 

FTR 

FTR 

%  >  6  GPS 

FTR 

FTR 

FTR 

Table  7. 

Nominal  vs  Degraded  Confidence  Interval  Comparison 

Scenario  Comparison 

Metric 

Degrad-0  -  Nom-0 

Degrad-5  -  Nom-5 

Degrad-10  -  Nom-10 

Pm 

-0.28%  ±  0.27% 

-0.20%  ±  0.26% 

0.04%  ±  0.17% 

Duration  (min) 

11.13  ±  7.97 

19.69  ±  7.81 

19.04  ±  13.97 

Casualties 

6.92  ±  9.14 

8.68  ±  7.21 

-0.44  ±  3.88 

Engagements 

2.68  ±  2.34 

4.08  ±  2.83 

0.96  ±  1.28 

%  Time  Lost 

0.01%  ±  0.05% 

0.01%  ±  0.07% 

0.01%  ±  0.09% 

%  <  4  GPS 

-0.00%  ±  0.04% 

0.01%  ±  0.08% 

-0.05%  ±  0.27% 

%  >  6  GPS 

0.03%  ±  0.26% 

0.05%  ±  0.27% 

0.08%  ±  0.18% 
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Table  8.  Nominal  vs  Num  Sats  Removed  P- Value  Comparison 


Metric 

Norn-0  vs  Nom-5 

Scenario  Comparison 

Nom-0  vs  Nom-10  Nom-5  vs  Nom-10 

-P50 

FTR 

Reject 

Reject 

Duration  (min) 

0.05 

0.00 

0.00 

Casualties 

0.57 

0.00 

0.00 

Engagements 

0.14 

0.00 

0.00 

%  Time  Lost 

FTR 

FTR 

FTR 

%  <  4  GPS 

FTR 

Reject 

Reject 

%  >  6  GPS 

FTR 

Reject 

Reject 

Table  9.  Nominal  vs  Num  Sats  Removed  Confidence  Interval  Comparison 


Metric 

Nom-5  -  Nom-0 

Scenario  Comparison 

Nom-10  -  Nom-0 

Nom-10  -  Nom-5 

P50 

-0.24%  ±  0.27% 

-0.60%  ±  0.21%  ■ 

-0.36%  ±  0.22% 

Duration  (min) 

7.52  ±7.40 

47.31  ±  11.18 

39.79  ±  9.38 

Casualties 

3.00  ±  10.86 

16.68  ±  8.21 

13.68  ±  6.59 

Engagements 

2.20  ±  3.00 

5.80  ±  2.28 

3.60  ±  2.07 

%  Time  Lost 

0.01%  ±  0.05% 

0.02%  ±  0.06% 

0.01%  ±  0.07% 

%  <  4  GPS 

0.01%  ±  0.05% 

0.45%  ±  0.20% 

0.44%  ±  0.20% 

%  >  6  GPS 

-0.09%  ±  0.27% 

-0.56%  ±  0.21%  ■ 

-0.47%  ±  0.22% 
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Table  10.  Degraded  vs  Num  Sats  Removed  P-Value  Comparison 


Metric 

Degrad-0  vs  Degrad-5 

Scenario  Comparison 

Degrad-0  vs  Degrad- 10 

Degrad-5  vs  Degrad- 10 

Pm 

FTR 

Reject 

FTR 

Duration  (min) 

0.00 

0.00 

0.00 

Casualties 

0.11 

0.01 

0.02 

Engagements 

0.00 

0.00 

0.53 

%  Time  Lost 

FTR 

FTR 

FTR 

%  <  4  GPS 

FTR 

Reject 

Reject 

%  >  6  GPS 

FTR 

Reject 

Reject 

Table  11.  Degraded  vs  Num  Sats  Removed  Confidence  Interval  Comparison 


Metric 

Degrad-5 

-  ] 

Degrad-0 

Scenario  Comparison 

Degrad- 10  -  Degrad-0 

Degrad- 10  - 

Degrad-5 

Pm 

-0.16% 

± 

0.25% 

-0.28% 

± 

0.23% 

-0.12% 

± 

0.21% 

Duration  (min) 

16.08 

± 

6.67 

55.22 

± 

12.45 

39.14 

± 

11.63 

Casualties 

4.76 

± 

5.96 

9.32 

± 

6.30 

4.56 

± 

3.64 

Engagements 

3.60 

± 

2.38 

4.08 

± 

1.98 

0.48 

± 

1.55 

%  Time  Lost 

0.01% 

± 

0.07% 

0.02% 

± 

0.08% 

0.01% 

± 

0.09% 

%  <  4  GPS 

0.02% 

± 

0.07% 

0.40% 

± 

0.20% 

0.38% 

± 

0.20% 

%  >  6  GPS 

-0.08% 

± 

0.27% 

-0.51% 

± 

0.23% 

-0.43% 

± 

0.24% 

Initial  Insights. 

The  first  broad  trend  observed  across  the  p-value  comparison  tables  is  the  fewer 
number  of  significant  p-values  when  comparing  the  nominal  and  degraded  operations 
with  the  same  number  of  satellites  removed.  The  trend  shows  that  the  model  is 
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more  sensitive  to  the  number  of  satellites  lost  for  a  given  operation  configuration 
versus  strictly  the  difference  between  the  nominal  and  degraded  operations.  This  is 
an  intuitive  result,  as  the  difference  between  the  nominal  and  degraded  operations  is 
not  severe,  but  the  removal  of  multiple  satellites  should  drastically  impact  the  model 
and  create  significant  differences  in  the  metrics.  The  insight  gained  is  useful  moving 
forward  and  provides  verification  to  the  model  acting  as  anticipated  under  specific 
configurations. 

The  second  broad  trend  is  that  the  percentage  of  time  lost  is  continually  insignif¬ 
icant  across  all  scenarios.  Additionally,  the  point  estimator  and  half  widths  observed 
in  Tables  7,  9,  and  11  are  small  in  magnitude.  As  a  result,  it  is  possible  to  con¬ 
clude  that  GPS  constellation  degradation  is  not  significantly  impacting  the  overall 
percentage  of  time  in  which  the  SOF  team  is  lost. 

Within  the  specific  metrics  there  are  several  trends  which  are  nearly  significant  in 
every  scenario.  The  first  is  mission  duration  which  is  always  significantly  different  and 
always  increases  in  value  with  increased  degradation.  Tables  7,  9,  and  11  also  show 
that  there  is  a  large  difference  between  scenarios  as  indicated  by  the  point  estimators 
in  the  confidence  intervals  and  half  widths.  This  helps  support  the  conclusion  of 
practically  as  well  as  statistically  difference  in  the  metrics.  A  key  insight  attained  from 
the  trend  is  that  the  overall  mission  duration  is  highly  sensitive  to  GPS  performance. 
If  the  WMD  needs  to  be  removed  within  a  small  time  window,  having  the  highest 
performing  GPS  constellation  would  be  critical. 

Two  often  significant  metrics  which  provide  similar  information  are  the  percentage 
of  time  less  than  four  GPS  and  greater  than  six  GPS  satellites  are  available.  As 
satellites  are  removed  the  percentage  of  time  less  than  four  GPS  satellites  are  available 
tends  to  increase  while  the  percentage  of  time  greater  than  six  GPS  satellites  are 
available  tends  to  decrease.  It  is  important  to  note  that  the  significant  difference 
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only  occurs  when  the  nominal  or  degraded  scenarios  are  compared  with  removing  ten 
satellites.  This  is  an  intuitive  result  as  removing  a  large  portion  of  the  initial  ten 
satellites  in  view  reduces  the  overall  number  of  satellites  available  over  the  duration 
of  the  mission,  ft  is  also  interesting  to  note  that  the  point  estimator  and  half  width 
for  the  comparison  between  five  and  ten  satellites  removed  is  nearly  identical  to  the 
statistics  from  zero  to  ten  satellites  removed.  As  a  result,  it  is  possible  to  conclude 
that  the  majority  of  the  impact  from  removing  satellites  occurs  after  five  satellites 
have  been  removed. 

Another  key  insight  gained  from  both  metrics  is  that  the  GPS  constellation  is 
not  able  to  quickly  recover  from  a  large  number  of  lost  satellites  with  the  remaining 
satellites  in  orbit.  Even  though  the  expansive  GPS  constellation  contains  many  satel¬ 
lites  in  several  different  plains,  the  originally  non-impacted  satellites  will  not  reach 
the  area  of  interest  in  time  to  restore  capability.  This  is  primarily  due  to  the  short 
duration  of  the  selected  mission  of  under  a  few  hours.  In  reality,  a  GPS  satellite  takes 
12  hours  to  complete  one  orbit  around  the  Earth  which  is  not  fast  enough  to  provide 
near  instantaneous  resiliency  to  the  constellation.  Note  in  this  study  we  do  not  con¬ 
sider  activating  spare  satellites  or  launching  additional  satellites  into  the  constellation 
as  a  responsive  tactic  referenced  in  the  Literature  Review.  As  a  result  of  the  insight 
gained,  the  percentage  of  time  less  than  four  GPS  satellites  are  available  is  retained 
which  represents  the  more  crucial  capability  of  three  dimensional  geolocation. 

A  further  specific  insight  is  connected  to  the  decrease  in  P50  with  an  increase  in 
casualties  and  number  of  engagements.  The  evidence  is  located  in  Table  8  where 
there  is  a  difference  between  metrics  with  a  larger  number  of  satellites  removed. 
Interestingly,  there  is  not  a  statistically  significant  difference  between  zero  and  five 
satellite  removals  which  may  be  an  indication  of  an  non  linear  increase  in  the  metrics 
with  increased  satellite  removals.  Similar  evidence  is  shown  in  Table  10,  but  it  is 
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not  as  convincing.  Overall,  the  observed  trend  supports  intuition  as  an  increased 
number  of  engagements  should  lead  to  more  casualties  which  reduces  the  chance  of 
losing  less  than  25  soldiers.  Tables  9  and  11  also  show  that  the  mean  values  for  the 
number  of  casualties  and  engagements  also  increase  as  the  number  of  satellites  are 
removed.  The  significant  changes  in  the  metrics  are  all  indications  that  a  reduced 
GPS  constellation  is  degrading  mission  performance.  As  a  result,  the  number  of 
casualties  and  engagements  are  retained  for  further  analysis. 

Several  important  insights  are  developed  from  the  initial  perspective  on  the  data. 
First,  is  that  mission  duration  is  highly  sensitive  to  GPS  degradation  while  the  per¬ 
centage  of  time  lost  is  not  sensitive.  Furthermore,  the  percentage  of  time  less  than  four 
GPS  satellites  and  the  percentage  of  time  greater  than  six  GPS  satellites  are  avail¬ 
able  are  intuitively  impacted  by  a  reduced  number  of  GPS  satellites  which  presents 
an  initial  conclusion  on  the  inherent  lack  of  resiliency  of  the  GPS  constellation  over 
short  mission  durations.  Additionally,  the  connection  between  P50  to  number  of  ca¬ 
sualties  and  engagements  is  displayed  throughout  the  p- value  and  confidence  interval 
comparisons.  As  a  result,  four  metrics  provide  insight  into  the  model  which  include 
mission  duration,  number  of  casualties,  number  of  engagements,  and  percentage  of 
time  less  than  four  GPS  satellites  are  available.  Each  of  the  four  metrics  receives  fur¬ 
ther  analysis  in  order  to  gain  additional  insight  into  the  GPS  constellation  resiliency. 

4.4  Full  Production  Runs 

The  full  experiment  involves  the  remaining  data  from  the  additional  scenarios. 
The  objective  is  to  have  data  from  every  level  of  satellite  removal  from  zero  to  ten  for 
both  the  nominal  and  degraded  operations.  Table  12  depicts  the  run  configurations 
required  to  collect  all  of  the  necessary  data. 
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Table  12.  Full  Production  Runs 


Factor 

Levels 

Design  Points 

Total  Reps 

Operations 

2 

22 

550 

Number  of  Sats  Removed 

11 

4.5  Graphical  Insight 

The  JMP  statistical  package  is  used  for  data  analysis.  Compelling  insight  and 
a  greater  understanding  of  the  model  outputs  is  achieved  by  observing  several  plots 
of  the  metrics  with  respect  to  the  number  of  satellites  removed.  Additionally,  the 
replications  are  grouped  based  upon  their  type  of  operations  as  either  nominal  or 
degraded.  The  distinct  patterns  are  interpretted  and  retained  for  further  analysis. 
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Mission  Duration  vs  Number  of  Satellites  Removed. 


The  first  graph  of  interest  plots  the  individual  95%  confidence  intervals  of  the 
mission  duration  as  the  number  of  satellites  removed  increases  as  shown  in  Figure  6. 
The  confidence  intervals  show  a  generally  increasing  trend  in  mission  duration  as  the 
number  of  satellites  removed  increase  regardless  of  the  type  of  operations.  A  key  ob¬ 
servation  from  the  “Degraded”  section  of  the  graph  displays  a  distinct  jump  between 
five  and  six  satellites  removed  where  the  confidence  intervals  no  longer  overlap.  This 
is  an  indication  of  a  statistically  longer  mission  duration  once  the  GPS  constellation 
loses  six  satellites. 

MeanfMission  Duration)  vs.  Number  of  Satellites  Removed  •  Mean(Mission  Duration) 

Operations 

Degraded  Nominal 

150-  -r 

140- 

130-  J_ 

120- 

C  I  < 


0123456789  10  0123456789  10 
Number  of  Satellites  Removed 

Figure  6.  Mission  Duration  Individual  95%  Confidence  Interval  Graph 
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Number  of  Engagements  vs  Number  of  Satellites  Removed. 

A  similar  and  more  abrupt  pattern  is  observed  in  the  “Degraded”  portion  of  the 
individual  95%  confidence  interval  graph  for  the  number  of  engagements  shown  in 
Figure  7.  The  jump  occurs  between  four  and  five  satellites  removed  which  is  similar 
to  the  threshold  noted  from  mission  duration.  An  important  insight  gained  is  that 
there  is  a  statistically  significant  increase  in  the  number  of  engagements  if  the  GPS 
constellation  loses  more  than  four  satellites. 

Mean(  Number  of  Engagements)  vs.  Number  of  Satellites  Removed  •  Mean(Numberof  Engagement) 

Operations 

Degraded  Nominal 

12-  j  _  T  T 

10- 


4- 

2- 

0123456789  10  0123456789  10 
Numberof  Satellites  Removed 

Figure  7.  Number  of  Engagements  Individual  95%  Confidence  Interval  Graph 

Number  of  Casualties  vs  Number  of  Satellites  Removed. 

Another  response  providing  insight  are  the  individual  95%  confidence  intervals 
associated  with  the  number  of  casualties  as  shown  in  Figure  8.  The  graphic  shows  an 
increasing  trend  in  the  number  of  casualties  as  the  scenario  becomes  more  degraded. 
The  key  observation  from  the  “Degraded”  portion  of  Figure  8  is  the  narrowing  of  the 
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confidence  interval  after  removing  four  satellites.  A  smaller  range  of  the  confidence 
interval  is  indicative  of  a  reduction  in  variation  from  the  replications.  At  the  moment 
it  is  not  obvious  what  is  causing  the  variance  reduction;  however,  it  is  important  to 
infer  that  there  is  a  consistently  high  number  of  casualties  once  a  large  number  of 
satellites  have  been  removed. 


Mean(Number  of  Casualties)  vs.  Number  of  Satellites  Removed 

Operations 

Degraded  Nominal 


MeanfNumber  of  Casualties) 


7R  25 


20 


0123456789  10  0123456789  10 
Numberof  Satellites  Removed 


Figure  8.  Number  of  Casualties  Individual  95%  Confidence  Interval  Graph 


Percentage  of  GPS  Satellites  Available  vs  Number  of  Satellites  Re¬ 
moved. 

Observing  the  data  output  for  the  percentage  of  time  less  than  four  GPS  satellites 
are  available  provides  information  into  the  exponential  nature  of  the  response  variable 
as  shown  in  Figure  9.  The  output  data  is  displayed  instead  of  the  confidence  intervals 
due  to  the  inability  to  automatically  display  the  correct  proportion  confidence  inter¬ 
vals  required  for  the  metric.  Nevertheless,  the  output  data  does  display  results  which 
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provide  insight  into  the  response.  The  data  remains  fairly  constant  until  more  than 
seven  satellites  are  removed  at  which  point  the  percentage  of  time  in  which  less  than 
four  GPS  satellites  are  in  view  increases  dramatically.  A  similar  graph  is  displayed 
from  the  data  output  from  the  percentage  of  time  greater  than  six  GPS  satellites  are 
available  shown  in  Figure  10.  The  threshold  now  occurs  around  five  to  six  satellites 
removed  and  displays  a  dramatic  decrease  in  the  response  variable.  Both  of  the  re¬ 
ponses  are  highly  dependent  upon  the  number  of  number  of  satellites  available.  If 
too  many  satellite  are  removed  then  it  is  increasingly  likely  to  register  high  values  for 
the  percentage  of  time  there  are  fewer  than  four  satellites  available  and  low  values  for 
the  percentage  of  time  that  there  are  greater  than  six  satellites  available. 
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%  <  4  GPS 


%  <  4  GPS  vs.  Number  of  Satellites  Removed 


Operations 


Figure  9.  Percent  of  Time  Less  Than  Four  GPS  Satellites  Available  Output  Data 
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%  >  6  GPS  vs.  Number  of  Satellites  Removed 

Operations 

Degraded  Nominal 


Figure  10.  Percent  of  Time  Greater  Than  Six  GPS  Satellites  Available  Output  Data 

P50  vs  Number  of  Satellies  Removed. 

Figure  11  provides  the  mean  value  for  P50  when  compared  with  the  number  of 
satellites  removed.  Similar  to  the  output  data  shown  in  Figures  9  and  10,  the  mean 
value  is  the  most  accessible  and  yet  still  insightful  perspective  on  P50.  As  evident 
by  both  type  of  operations,  P50  is  negatively  impacted  by  an  increasing  number  of 
satellites  removed.  When  large  numbers  of  satellites  are  removed,  it  appears  as  though 
there  is  a  small  probability  of  completing  the  mission  with  fewer  than  25  casualties. 
Additionally,  the  low  starting  point  for  the  “Degraded”  operations  even  when  none 
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of  the  satellites  have  been  removed  is  an  important  disparity  worth  noting. 

Mean(P_50)  vs.  Number  of  Satellites  Removed  •  Mean(P_50) 
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Figure  11.  -P50  Mean  Value  Graph 


Mean(P_50)  vs.  Number  of  Satellites  Removed 
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4.6  DOE 

A  designed  experiment  is  performed  to  determine  how  the  factors  effect  the  re¬ 
sponse  variables.  Due  to  the  minimal  resources  required  to  complete  one  replication,  a 
full  factorial  design  is  utilized  to  provide  un-aliased  analysis  between  the  levels  of  the 
factors.  Insights  are  gained  from  an  effects  model  which  focuses  on  each  factor  along 
with  the  interaction  between  the  factors.  The  results  are  analyzed  in  the  two-factor 
analysis  of  variance  (ANOVA)  table  to  determine  the  significance  of  each  factor.  [23] 
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Responses. 


The  responses  under  consideration  are  the  mission  duration,  number  of  casualties, 
number  of  engagements,  and  the  percentage  of  time  less  than  four  GPS  satellites  are 
available,  as  identified  as  being  sensitive  to  the  factors  in  the  initial  analysis.  Mission 
duration  is  classified  as  a  continuous  response  where  as  casualties  and  engagements  are 
integer  variables.  The  percentage  of  time  less  than  four  GPS  satellites  are  available 
is  also  unique  in  that  it  is  continuous,  but  is  bounded  between  0  and  1.  Each  of  the 
responses  are  analyzed  through  the  ANOVA. 

Factors. 

There  are  two  factors  used  in  the  model  to  include  the  type  of  operations  and  the 
number  of  satellites  removed.  The  type  of  operations  is  a  nominal  variable  composed 
of  two  levels;  nominal  or  degraded  operations  as  described  in  Chapter  3.  The  number 
of  removed  satellites  from  the  model  is  the  second  factor  which  is  also  considered 
nominal  and  has  11  different  levels  to  reflect  the  integer  value  of  satellites  removed 
from  zero  to  ten.  The  full  factorial  run  configuration  requires  22  different  design 
points  to  cover  all  combinations  of  the  factors. 

ANOVA. 

The  ANOVA  is  performed  on  each  response  variable  by  using  both  factors  and 
their  interaction.  Upon  further  inspection,  the  addition  of  the  interaction  variable 
between  the  factors  does  not  provide  any  significant  benefit  to  the  model.  Instead, 
the  marginal  gain  in  model  statistics  with  the  interaction  term  is  not  worth  over 
fitting  the  model  with  a  new  set  of  variables.  Furthermore,  with  the  exception  of  the 
percentage  of  time  less  than  four  GPS  satellites  are  available,  the  lack  of  fit  test  did 
not  reject  the  null  hypothesis  indicating  that  an  interaction  term  is  not  necessary. 
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As  a  result,  all  of  the  regression  models  are  restricted  to  using  the  single  factors  as 
effects. 

Key  results  from  the  ANOVA  include  the  significance  of  the  overall  model  along 
with  the  individual  factors.  Furthermore,  the  numerical  impact  of  each  factor  and 
factor  level  is  estimated  to  provide  specific  insight  into  the  impact  on  the  response. 
Additionally,  using  Tukey’s  test  allows  for  the  statistical  pairwise  comparison  be¬ 
tween  the  levels  of  each  factor  providing  more  insight  into  any  large  changes  in  the 
responses.  ANOVA  enables  insight  into  the  specific  situations  which  most  impact 
GPS  constellation  resiliency. 

Assumption  Verification. 

There  are  four  assumptions  to  check  before  interpreting  the  results  from  the 
ANOVA.  The  assumptions  assess  the  residuals  of  the  ANOVA  model  on  their  normal¬ 
ity,  independence,  constant  variance  against  predicted  values,  and  constant  variance 
against  the  variables  of  the  model.  The  residuals  exhibit  normality  for  all  of  the 
ANOVA  models  as  shown  by  a  histogram  and  normal  probability  plots  of  the  residu¬ 
als  shown  in  the  Appendix  A.  One  minor  normality  violation  may  be  present  in  the 
percentage  of  time  less  than  fonr  GPS  satellites  are  available  model  where  the  major¬ 
ity  of  the  residuals  are  contained  near  the  center  of  the  distribution.  Independence 
of  the  residuals  is  not  directly  assessed  in  the  study,  however,  we  can  confidently 
conclude  that  each  replication  is  initiated  with  its  own  random  number  and  that  the 
simulation  is  not  dependent  upon  the  results  from  the  previous  replication.  The  resid¬ 
uals  versus  predicted  plots  yield  some  concern  as  far  as  non-constance  variance  for  all 
of  the  ANOVA  models,  however,  no  transformations  are  performed  on  the  responses. 
The  non-constant  variance  does  not  appear  to  be  severe  enough  to  warrant  the  loss  of 
interpret-ability  from  a  transformation.  Furthermore,  almost  all  of  the  residual  versus 
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variable  plots  exhibit  constant  variance  with  the  except  of  the  percentage  of  time  less 
than  four  GPS  satellites  are  available  model  where  the  residuals  appear  to  fan  out¬ 
ward  with  an  increasing  number  of  removed  satellites.  Overall,  the  model  residuals 
meet  all  of  the  required  assumptions  for  the  AN OVA  models  and  the  respective  plots 
are  displayed  in  the  Appendix  A. 

Summary  Statistics. 

The  ANOVA  models  are  generated  in  JMP  and  the  parameter  estimates  are  avail¬ 
able  in  the  appendix.  Summary  statistics  from  each  of  the  models  are  displayed  in 
Table  13. 


Table  13.  ANOVA  Model  Statistics  w/o  Interaction  Term 


Summary  Statistics 

Duration 

Response 

Casualties  Engagement 

%  <  4  GPS 

R2 

0.577 

0.255 

0.327 

0.922 

R2- adj 

0.568 

0.24 

0.314 

0.92 

MSE 

385.4 

158.36 

15.593 

0.002 

Overall  F 

<  .0001 

<  .0001 

<  .0001 

<  .0001 

Lack  of  Fit 

0.0627 

0.0771 

0.4188 

<  .0001 

Operations 

<  .0001 

<  .0001 

<  .0001 

0.0988 

Sats  Removed 

<  .0001 

<  .0001 

<  .0001 

<  .0001 

All  of  the  generated  models  are  significant  due  to  the  results  from  the  overall 
model  F  test,  however  several  models  explain  a  larger  proportion  of  the  total  variance 
in  the  data.  The  R 2  and  R2- adj  statistics  assess  the  proportion  of  the  total  variance 
explained  by  the  specific  model.  The  results  show  that  the  mission  duration  and 
percentage  of  time  less  than  four  GPS  satellites  are  available  models  explain  a  larger 
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proportion  of  the  total  variance  than  the  number  of  casualties  and  engagements.  As  a 
result,  it  is  possible  to  conclude  that  mission  duration  and  percentage  of  time  less  than 
four  GPS  satellites  are  available  are  more  sensitive  to  the  factors  than  the  number 
of  casualties  and  engagements.  Intuition  supports  the  high  R 2  and  R2- adj  statistics 
dependency  of  the  percentage  of  time  less  than  four  GPS  satellites  are  available  model 
due  to  the  inherent  relationship  between  removing  satellites  from  the  model  and  the 
possibility  of  having  fewer  satellites  in  view.  Additionally,  as  mentioned  above  all  of 
the  models  except  percentage  of  time  less  than  four  GPS  satellites  are  available  fail 
to  reject  the  lack  of  fit  test  demonstrating  that  interaction  terms  are  not  required  and 
that  the  models  are  properly  specified. 

Both  operations  and  the  number  of  satellites  removed  are  significant  factors  in  the 
models.  One  exception  occurs  for  the  percentage  of  time  less  than  four  GPS  satellites 
are  available  response  where  operations  is  not  significant  indicating  that  the  type  of 
operations  does  not  significantly  impact  the  metric.  Aside  from  the  one  exception, 
there  is  compelling  evidence  to  suggest  that  the  factors  arc  highly  significant  variables 
toward  defining  the  metrics.  As  a  result,  it  is  possible  to  conclude  that  adequately 
controlling  the  type  of  operations  and  the  number  of  satellites  removed  is  essential 
for  improving  satellite  resiliency. 

Pairwise  Comparison  -  Tnkey  Test. 

To  gain  further  insight  into  the  ANOVA  models,  the  Tukey  test  is  used  to  perform 
a  pair-wise  comparison  of  the  levels  of  the  number  of  satellites  removed.  The  mission 
duration  model  demonstrates  that  there  are  several  important  locations  in  terms  of 
the  number  of  satellites  removed.  The  results  from  the  Tukey  test  are  shown  in 
Figure  12. 
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LS  Means  Plot 


Figure  12.  Mission  Duration  Tukey  Test  Results 


There  are  several  unique  groupings  displayed  in  the  Tukey  test  results  for  the  du¬ 
ration  model.  The  first  grouping  displays  that  removing  eight,  nine,  or  ten  satellites 
produces  statistically  similar  outputs  that  are  statistically  different  from  the  remain¬ 
ing  levels  of  satellites  removed.  Other  groupings  occur  for  six  and  seven  satellite 
removals  along  with  the  remaining  zero  through  five  satellite  removals.  Each  of  the 
groupings  indicate  specific  threshold  values  for  the  levels  that  will  generate  the  same 
response.  For  example,  the  GPS  constellation  could  lose  from  zero  to  five  satellites 
and  experience  the  same  impact  to  mission  duration.  However,  once  a  sixth  satellite  is 
removed,  then  the  GPS  constellation  experiences  degradation  leading  to  an  extended 
missions  duration  above  100  minutes  on  average.  If  more  than  seven  satellites  are 
removed,  then  mission  duration  increase  to  120  minutes  on  average.  As  a  result,  the 
different  threshold  levels  provide  information  into  satellite  resiliency. 

Similar  insight  is  attained  from  the  Tukey  plots  from  each  of  the  ANOVA  models. 
The  results  from  the  Tukey  test  for  the  number  of  casualties  response  are  shown  in 
Figure  13. 
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LS  Means  Plot 


Figure  13.  Casualties  Tukey  Test  Results 

The  most  important  shift  observed  in  Figure  13  is  the  jump  around  five  satellites 
removed.  The  grouping  of  satellite  removals  from  zero  to  four  reports  below  20 
casualties  on  average  whereas  the  group  from  six  to  ten  report  near  30  casualties.  As 
a  result,  the  threshold  level  is  the  removal  of  five  satellites  where  the  GPS  constellation 
is  resilient  in  terms  of  the  number  of  casualties. 

The  results  from  the  Tukey  test  for  the  number  of  engagements  response  are  shown 
in  Figure  14. 


LS  Means  Plot 

12- 

CO 

c  co  -i  n- 
<D  c  1 U 

E 

Si  8- 
£3  6- 

LU 

4- 

H 

H 

H 

F-1  1  '  2  '  3  '  4  '  5  '  6  '  7  '  F-1  9  '  10 


NumberofSatellites  Removed 
Figure  14.  Engagements  Tukey  Test  Results 

There  are  two  distinct  groups  identified  in  the  Tukey  test  with  the  threshold 
located  by  removing  five  satellites.  The  average  number  of  engagements  appear  to 
double  in  mean  value  once  more  than  four  satellites  are  removed.  From  the  insight 
gained  it  is  possible  to  conclude  that  the  GPS  constellation  exhibits  resiliency  until 
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more  than  four  satellites  are  removed.  As  a  result,  maintaining  the  performance  of 
the  GPS  constellation  is  achievable  by  ensuring  that  the  number  of  satellites  removed 
does  not  exceed  four. 

The  results  from  the  Tukey  test  for  the  percentage  of  time  less  than  four  GPS 
satellites  are  available  response  are  shown  in  Figure  15. 

LS  Means  Plot 

0.4 

CO  w 

Q-  C  0  3 
O  to  u'° 

—  TO 

?  2  0.2 
s?  ^  oi 
0 


As  displayed  in  Figure  15,  the  width  on  the  confidence  interval  is  small  with 
relation  to  the  other  Tukey  plots.  Additionally,  removing  zero  through  five  satellites 
does  not  generate  a  significant  difference  in  the  percentage  of  time  less  than  four  GPS 
satellites  are  available.  However,  every  additional  satellite  removed  creates  a  different 
output  for  the  response.  As  a  result,  the  threshold  value  for  resiliency  is  located  at  five 
satellites  removed.  Additionally,  it  is  important  to  note  that  mitigating  the  number 
of  satellites  removed  beyond  five  is  important  as  each  satellite  removed  creates  a 
significant  difference.  Also  worth  consideration  is  a  large  jump  between  seven  and 
eight  satellite  removals  where  the  percentage  of  time  less  than  four  GPS  satellites  are 
available  triples  in  value.  Avoiding  the  loss  of  an  extreme  number  of  GPS  satellites 
improves  overall  GPS  constellation  resiliency. 

Each  of  the  Tukey  plots  provide  unique  insight  into  the  levels  of  the  number  of 
satellites  removed  factor  and  help  identify  the  critical  thresholds  for  assessing  satellite 
resiliency.  Across  all  of  the  Tukey  plots,  it  appears  as  though  losing  more  than  Eve 


Figure  15.  %  <  4  GPS  Tukey  Test  Results 
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GPS  satellites  leads  to  significant  differences  in  the  response  variable.  If  the  number 
of  satellites  lost  can  be  kept  below  five,  then  it  is  possible  to  maintain  response  values 
that  are  similar  to  a  scenario  with  zero  satellites  lost.  The  ability  to  operate  in  a 
degraded  state  with  a  number  of  satellites  removed  and  still  provide  the  initial  metric 
values  indicates  that  the  GPS  constellation  is  resilient.  As  a  result,  it  is  possible  to 
conclude  that  the  GPS  constellation  is  resilient  up  to  the  removal  of  five  satellites 
from  the  initial  configuration. 

4.7  Conclusion 

Chapter  4  performs  the  analysis  and  interprets  the  results  as  they  pertain  to  GPS 
constellation  resiliency.  The  initial  analysis  is  used  to  gain  a  first  perspective  into 
the  output  data  from  a  series  of  design  points.  The  p-values  from  the  Paired  T  test 
and  confidence  intervals  reveal  that  mission  duration,  number  of  casualties,  number 
of  engagements,  and  the  percentage  of  time  less  than  four  GPS  satellites  are  available 
are  sensitive  to  the  factors.  Furthermore,  an  increased  number  of  satellites  removed 
appears  to  increase  the  sensitivity  of  the  metrics  and  create  statistically  significant 
differences.  Completing  the  remaining  production  runs  enables  the  graphical  analysis 
of  the  output  data.  Several  of  the  confidence  interval  graphs  display  trends  along  with 
threshold  values  to  statistically  significant  differences  in  the  metric.  Additionally, 
graphs  of  the  data  as  satellites  are  removed  show  exponential  trends  in  the  number 
the  satellites  available  along  with  a  sharp  decrease  in  the  P50  metric.  Finally,  a  formal 
ANOVA  model  for  each  response  variable  is  generated  which  provides  insight  into  the 
significance  of  the  effects.  In  general,  the  ANOVA  models  pass  the  assumption  tests 
along  with  providing  significance  p-values  for  each  of  the  factors.  The  most  compelling 
takeaway  from  Chapter  4  are  the  results  from  the  pairwise  Tukey  Test  between  the 
levels  of  the  number  of  satellites  removed.  It  is  evident  in  each  of  the  four  selected 


metrics  that  there  is  a  threshold  value  once  five  satellites  have  been  removed.  As 


soon  as  a  sixth  satellite  is  removed  the  metrics  register  a  significant  difference  in 
value.  As  a  result,  it  is  possible  to  conclude  that  the  GPS  constellation  provides 
the  highest  level  of  performance  and  exhibits  resiliency  until  five  satellites  have  been 
removed,  at  which  point  the  GPS  constellation  experiences  a  significant  decrease  in 
performance,  both  statistically  and  practically.  Chapter  5  contains  a  summary  of  the 
most  important  results  from  the  analysis  along  with  areas  for  further  research. 
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V.  CONCLUSIONS 


5.1  Overview 

Chapter  5  concludes  the  study  on  GPS  constellation  resiliency.  The  chapter  fo¬ 
cuses  on  the  contributions  provided  through  the  Methodology  along  with  a  summary 
of  the  important  insights  from  the  analysis.  Each  of  the  insights  provide  evidence 
that  supports  the  conclusion  on  the  GPS  constellation  resiliency.  Chapter  5  also 
details  areas  for  further  research  that  provide  more  insight  and  refinement  into  the 
assessment  of  satellite  resiliency.  The  chapter  concludes  with  a  conclusion  for  the 
study. 

5.2  Contributions  and  Analysis  Conclusions 

Assessing  satellite  resiliency  is  a  recent  topic  that  has  gained  traction  due  to  the 
increasing  availability  of  threats  and  our  increasing  dependency  on  space  assets.  Due 
to  the  new  emphasis,  there  are  only  a  few  studies  which  model  and  assess  satellite 
constellation  resiliency.  Our  research  specifically  focuses  on  the  resiliency  of  the  GPS 
constellation  in  an  urban  canyon  environment.  Additionally,  the  research  provides  a 
new  perspective  for  modeling  satellite  constellation  resiliency  by  using  the  agent-based 
SEAS  program  and  collecting  metrics  from  a  top  level  mission  viewpoint.  Traditional 
studies  on  the  GPS  constellation  have  a  technical  perspective  where  the  specific  GPS 
parameters,  such  as  position  dilution  of  precision  (PDOP),  are  compared  across  de¬ 
graded  scenarios.  In  reality,  however,  what  might  be  a  statistically  significant  dif¬ 
ference  between  the  PDOP  in  a  scenario  comparison,  might  not  actually  impact  the 
mission  in  which  the  GPS  constellation  is  supporting.  Instead,  by  using  an  agent- 
based  model  and  capturing  top  level  mission  metrics,  the  SOF  team  in  our  model  can 
stochastically  respond  to  a  degraded  GPS  constellation  to  determine  if  there  is  an 
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impact  to  the  overall  mission.  While  there  is  not  one  definitive  metric  for  resiliency 
as  noted  during  the  Literature  Review,  a  compilation  of  output  statistics  provide  in¬ 
sight  into  constellation  resiliency.  Each  of  the  selected  metrics  connect  back  to  the 
mission  priorities  which  include  minimizing  casualties,  duration,  engagements,  and 
the  percentage  of  time  less  than  four  GPS  satellites  are  available.  Furthermore,  the 
logic  within  the  SEAS  warfile  has  been  adapted  to  provide  degradation  to  the  GPS 
constellation  performance  with  the  removal  of  satellites.  As  a  result,  the  research 
provides  a  new  methodology  for  future  studies  into  satellite  constellation  resiliency. 

To  further  strengthen  the  study,  the  analysis  is  given  statistical  rigor  through 
difference  hypothesis  tests  and  confidence  intervals.  The  initial  analysis  reveals  that 
the  model  is  sensitive  to  a  large  increase  in  the  number  of  satellites  removed  from 
the  simulation.  Furthermore,  it  appears  as  though  the  majority  of  the  impact  to  the 
metrics  occurs  between  five  and  ten  satellites  removed  which  indicates  that  degraded 
GPS  performance  may  have  an  non  linear  impact  with  an  increase  in  satellites  re¬ 
moved.  It  is  also  evident  from  the  full  production  run  data  that  mission  duration 
is  highly  impacted  by  removing  satellites  as  shown  in  Figure  6.  Additionally,  the 
mission  duration  is  also  increased  in  the  “Degraded”  operations.  Similar  results  are 
shown  with  the  number  of  engagements  and  casualties  in  Figures  7  and  8  respectfully. 
Another  important  conclusion  is  the  large  disparity  between  the  type  of  operations 
and  the  low  values  for  the  P50  metric  as  shown  in  Figure  11.  All  of  the  initial  sta¬ 
tistical  conclusions  confirm  intuition  that  the  top  level  mission  statistics  are  highly 
impacted  by  a  large  loss  of  GPS  satellites. 

The  designed  experiment  and  ANOVA  also  support  the  conclusion  that  reducing 
the  number  of  GPS  satellites  significantly  impacts  the  response  variables.  Each  of  the 
ANOVA  results  are  summarized  in  Table  13  which  shows  that  the  mission  duration 
and  percentage  of  time  that  there  are  less  than  four  GPS  satellites  available  are  the 
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strongest  models  in  terms  of  K2- adj.  Furthermore,  it  appears  as  though  the  number 
of  satellites  removed  is  a  significant  factor  for  all  of  the  responses.  The  key  insight 
gained  from  the  ANOVA  models  is  found  in  the  series  of  Tukey  tests  on  the  number 
of  satellites  removed  for  each  of  the  models.  Tukey’s  test  clearly  indicates  a  threshold 
value  near  five  satellites  removed  where  the  metrics  experience  a  significant  shift 
in  response  values.  The  significant  difference  indicates  that  the  GPS  constellation  is 
exhibiting  resiliency  even  though  several  satellites  have  been  removed  from  the  model. 
As  a  result,  the  designed  experiment  provides  further  support  for  the  initial  analysis 
along  with  the  most  compelling  insights  into  satellite  resiliency. 

The  analysis  suggests  that  the  GPS  constellation  in  our  scenario  does  exhibit  re¬ 
siliency  in  a  degraded  urban  canyon  environment.  The  ability  to  provide  statistically 
similar  model  output  metrics  indicates  that  even  though  the  GPS  specific  param¬ 
eters  are  degraded,  the  overall  mission  performance  is  not  impacted.  This  finding 
connects  directly  back  to  the  selected  definition  of  resiliency  from  AFSPC  which  is, 
“the  ability  of  a  system  architecture  to  continue  providing  required  capabilities  in 
the  face  of  system  failures,  environmental  challenges,  or  adversary  action”  [3].  We 
believe  the  resiliency  of  the  GPS  constellation  is  inherent  in  the  constellation  design 
which  has  nearly  a  dozen  satellites  initially  over  the  area  of  interest.  The  large  num¬ 
ber  of  satellites  over  the  area  of  interest  is  due  to  the  global  coverage  pattern  of  the 
constellation  which  provides  a  large  amount  of  overlap.  While  the  GPS  constellation 
does  exhibit  resiliency,  it  is  not  resilient  beyond  the  loss  of  five  satellites.  The  Tukey 
plots,  Paired  T  tests,  and  confidence  intervals  support  this  conclusion  by  indicat¬ 
ing  a  quickly  changing  negative  impact  in  the  response  metrics  after  more  than  five 
satellites  are  lost.  Additionally,  as  noted  in  the  initial  analysis,  if  nearly  all  of  the 
satellites  initially  available  are  removed  from  the  scenario,  then  the  GPS  constella¬ 
tion  does  not  provide  instantaneous  resiliency  for  a  short  mission  duration.  Instead, 
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a  longer  mission  would  be  able  to  recover  as  soon  as  the  remaining  GPS  satellites 
pass  over  the  area  of  interest.  Furthermore,  it  is  important  to  note  that  the  threshold 
value  near  five  satellites  lost  and  conclusions  developed  throughout  the  analysis  are 
heavily  dependent  upon  the  GPS  Unit  logic  in  the  SEAS  warhle.  As  a  result,  policy 
considerations  on  the  resiliency  of  the  GPS  constellation  require  more  specific  study. 
Nevertheless,  the  empirical  evidence  indicates  that  the  GPS  constellation  is  resilient. 

5.3  Future  Research 

There  are  several  areas  for  further  research  found  in  the  GPS  Unit  logic  structure, 
the  factors  included  in  the  designed  experiment,  the  geometry  of  the  the  GPS  satel¬ 
lites,  and  the  satellite  constellation  under  consideration.  It  is  important  to  remember 
that  this  study  is  purely  for  education  purposes  and  does  not  contain  any  sensitive 
parameters  which  limits  the  model  validity.  Additionally,  due  to  time  constants,  the 
current  study  did  not  permit  significant  changes  to  the  provided  SEAS  model.  As 
a  result,  there  are  several  areas  in  which  the  research  can  be  expanded  and  further 
developed. 

One  area  for  further  research  is  the  sensitivity  of  the  GPS  Unit  logic  structure  in 
the  SEAS  warhle.  We  developed  a  logic  structure  without  any  direct  references  that 
at  face  value  appears  reasonable.  However,  it  is  highly  likely  that  further  investigation 
into  the  accuracy  and  availability  of  the  GPS  constellation  will  provide  a  different  logic 
structure.  If  the  desire  is  to  keep  the  research  unclassified,  then  sensitivity  analysis 
can  be  performed  by  providing  a  series  of  different  logic  structures  to  the  GPS  Unit. 
Then  by  replicating  the  analysis  performed  in  this  study  over  each  of  the  new  logic 
structures  it  is  possible  to  provide  insight  into  the  impact  of  the  factors  along  with 
determining  a  resiliency  threshold  value.  Further  refinement  of  the  logic  structure  is 
essential  for  adding  more  credibility  to  the  study. 
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Another  area  for  further  research  includes  incorporating  additional  factors  to  the 
designed  experiment,  ft  was  evident  that  the  R 2  —  adj  values  did  not  perform  as 
desired  for  the  number  of  casualties  and  engagement  responses.  An  example  of  an 
additional  factor  is  examining  the  level  of  degradation  as  the  SOF  team  is  closer 
to  the  buildings.  While  additional  factors  appear  to  be  a  straight  forward  improve¬ 
ment,  selecting  factors  which  are  directly  connected  to  the  GPS  constellation  and 
implementing  them  into  the  warhle  is  a  challenge.  Nevertheless,  investigating  further 
factors  is  worth  the  added  effort  in  order  to  identify  the  most  important  pieces  for 
attaining  satellite  resiliency. 

As  noted  by  Bell’s  thesis  in  the  Literature  Review,  the  geometry  of  the  satellites 
is  important  to  GPS  capability  instead  of  just  the  number  of  satellites  in  view.  For 
example,  a  fewer  number  of  satellite  with  a  better  geometry  provides  better  accuracy 
for  the  GPS  reciever.  Currently  there  is  not  a  geolocation  algorithm  provided  in 
the  SEAS  warhle.  Implementation  of  the  geolocation  algorithm  may  be  challenging 
within  SEAS,  however,  successful  implementation  provides  a  more  accurate  model 
along  a  new  source  of  metrics  for  analysis.  We  highly  suggest  adding  a  geolocation 
algorithm  as  an  area  for  further  research. 

A  final  area  for  further  research  is  to  model  different  satellites  constellations.  The 
GPS  constellation  is  inherently  resilient  due  to  its  number  of  satellites  in  several 
difference  planes.  Other  constellations,  however,  are  not  as  diverse  and  may  not 
be  able  to  provide  the  same  capabilities  under  degraded  scenarios.  Furthermore, 
other  constellations  provide  unique  capabilities  which  requires  a  different  model  with 
different  metrics  to  assess  resiliency.  Nevertheless,  constellation  resiliency  is  crucial 
for  all  of  our  space  systems  and  is  worth  the  added  effort. 
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5.4  Conclusion 


Satellite  constellation  resiliency  is  an  important  consideration  moving  forward. 
Both  current  operational  decisions  and  future  programmatic  purchases  should  focus 
on  selecting  the  constellation  that  will  continue  to  provide  the  necessary  capabilities 
even  in  a  contested  environment.  The  methods  and  metrics  for  assessing  satellite 
resiliency  are  vital  to  ensuring  that  the  analysis  is  directed  properly  and  provides 
relevant  information.  We  have  provide  one  method  and  set  of  metrics  for  assessing 
the  resiliency  of  the  crucial  GPS  constellation  in  a  challenging  and  degraded  environ¬ 
ment.  While  our  results  do  indicate  that  the  GPS  constellation  exhibits  resiliency, 
the  required  capabilities  are  negatively  impacted  after  the  loss  of  multiple  satellites. 
Further  refinement  of  the  study  is  required  through  either  sensitivity  analysis  on 
the  logic  structure  or  a  more  complex  designed  experiment  in  order  to  provide  more 
definitive  guidance  to  decision  makers.  Also,  adding  a  geolocation  algorithm  is  also 
an  important  aspect  of  the  warble  for  strengthening  the  validity  of  the  model.  In 
addition,  the  methodology  can  be  applied  to  other  satellite  constellations  in  order  to 
reveal  the  hidden  strengths  and  weaknesses  of  our  space  assets.  Ensuring  resiliency 
for  all  of  our  satellite  constellations  is  necessary  for  current  and  future  operations  in 
an  effort  to  maintain  national  security. 
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Appendix  A.  ANOVA  ASSUMPTIONS  AND  MODELS 


1.1  Normal  Probability  Plots 


Distributions 


Figure  16.  Mission  Duration  Normal  Probability  Plot 
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Distributions 


Residual  Number  of  Casualties 


Figure  17.  Number  of  Casualties  Normal  Probability  Plot 
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Distributions 
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Figure  18.  Number  of  Engagements  Normal  Probability  Plot 


78 


Distributions 


Residual  %<4GPS 


0.66 


0.36 


0.02 


t5 

LL 

_a> 

c 

03 

3 

o 

m 

E 

i — 

o 


Figure  19.  %  <  4  GPS  Normal  Probability  Plot 
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1.2  Residual  vs  Predicted  Plots 


Figure  20.  Mission  Duration  Residual  vs  Predicted  Plot 


Figure  21.  Number  of  Casualties  Residual  vs  Predicted  Plot 
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Residual  by  Predicted  Plot 


Figure  22.  Number  of  Engagements  Residual  vs  Predicted  Plot 


Figure  23.  %  <  4  GPS  Residual  vs  Predicted  Plot 
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1.3  Residual  vs  Variable  Plots 


Oneway  Analysis  of  Residual 
Mission  Duration  By  Operations 
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Figure  24.  Mission  Duration  Residual  vs  Operations  Plot 


Oneway  Analysis  of  Residual  Mission 
Duration  By  Number  of  Satellites  Removed 
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Figure  25.  Mission  Duration  Residual  vs  Number  of  Satellites  Removed  Plot 
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Oneway  Analysis  of  Residual 
Number  of  Casualties  By  Operations 
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Figure  26.  Number  of  Casualties  Residual  vs  Operations  Plot 
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Oneway  Analysis  of  Residual  Number  of 
Casualties  By  Number  of  Satellites  Removed 
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Figure  27.  Number  of  Casualties  Residual  vs  Number  of  Satellites  Removed  Plot 
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Oneway  Analysis  of  Residual 
Number_of_Engagements  By  Operations 
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Figure  28.  Number  of  Engagements  Residual  vs  Operations  Plot 


Oneway  Analysis  of  Residual  Number_of_Engagements 
By  Number  of  Satellites  Removed 


Figure  29.  Number  of  Engagements  Residual  vs  Number  of  Satellites  Removed  Plot 
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Oneway  Analysis  of  Residual  %  <  4  GPS  By  Operations 
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Figure  30.  %  <  4  GPS  Residual  vs  Operations  Plot 


Oneway  Analysis  of  Residual  %  <  4  GPS  By  Number  of  Satellites  Removed 


Figure  31.  %  <  4  GPS  Residual  vs  Number  of  Satellites  Removed  Plot 
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1.4  Model  Statistics 


Mission  Duration  Model. 
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Figure  32.  Mission  Duration  Model  Statistics 


Number  of  Casualties  Model. 


Summary  of  Fit 

RSauare 

0.254932 

RSquare  Adi 

0.239698 

Root  Mean  Square  Error 

12.58402 

Mean  of  Response 

24.43455 

Observations  (or Sum  Wqts) 

550 

Analysis  of  Variance 

Sum  of 


Source 

DF 

Squares 

Mean  Square 

F  Ratio 

Model 

11 

29150.77 

2650.07 

16.7347 

Error 

538 

85196.37 

158.36 

Prob  >  F 

C.  Total 

549 

114347.14 

<.0001* 

Lack  Of  Fit 

Sum  of 


Source 

DF 

Squares 

Mean  Square 

F  Ratio 

Lack  Of  Fit 

10 

2660.935 

266.093 

1.7023 

Pure  Error 

528 

82535.440 

156.317 

Prob  >  F 

Total  Error 

538 

85196.375 

0.0771 

Max  RSq 

0.2782 


Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

24.434545 

0.536584 

45.54 

<  0001* 

OperationsrDeqradedl 

2.9690909 

0.536584 

5.53 

<  0001* 

NumberofSatellites  RemovedfOl 

-4.214545 

1.696829 

-2.48 

0.0133* 

Numberof  Satellites  Removeddl 

-7.874545 

1.696829 

-4.64 

<.0001* 

NumberofSatellites  Removed(21 

-8.454545 

1.696829 

-4.98 

<.0001* 

NumberofSatellites  Removed(31 

-7.634545 

1.696829 

-4.50 

<  0001* 

NumberofSatellites  Removedf41 

-5.574545 

1.696829 

-3.29 

0.0011* 

NumberofSatellites  Removed(51 

-0.334545 

1.696829 

-0.20 

0.8438 

NumberofSatellites  Removed(61 

4.3854545 

1.696829 

2.58 

0.0100* 

NumberofSatellites  Removedl71 

5.5454545 

1.696829 

3.27 

0.0012* 

NumberofSatellites  Removed(81 

7.5854545 

1.696829 

4.47 

<  0001* 

NumberofSatellites  Removed[91 

7.7854545 

1.696829 

4.59 

<  0001* 

Effect  Tests 

Sum  of 

Source 

Nparm 

DF 

Squares 

F  Ratio 

Prob  >  F 

Operations 

1 

1 

4848.525 

30.6176 

<.0001* 

NumberofSatellites  Removed 

10 

10 

24302.244 

15.3464 

<.0001* 

Figure  33.  Number  of  Casualties  Model  Statistics 
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Number  of  Engagements  Model. 
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Figure  34.  Number  of  Engagements  Model  Statistics 
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Figure  35.  Percentage  of  Time  <  4  GPS  Model  Statistics 


Appendix  B.  ACRONYMS 


ABMS  =  Agent-Based  Modeling  and  Simulation 

AFIT  =  Air  Force  Institute  of  Technology 

AFSPC  =  Air  Force  Space  Command 

ANOVA  =  Analysis  of  Variance 

ASAT  =  Anti-Satellite 

CAS  =  Complex  Adaptive  System 

DoD  =  Department  of  Defense 

EPA  =  Environmental  Protection  Agency 

GAO  =  Government  Accountability  Office 

GEO  =  Geosynchronous  Earth  Orbit 

GPS  =  Global  Positioning  System 

HEO  =  Highly  Elliptical  Orbit 

KPP  =  Key  Performance  Parameter 

LEO  =  Low  Earth  Orbit 

MOE  =  Measure  of  Effectiveness 

MOO  =  Measure  of  Outcome 

MOP  =  Measure  of  Performance 

MUA  =  Military  Utility  Analysis 

ORS  =  Operationally  Responsive  Space 

PDOP  =  Position  Dilution  of  Precision 

PNT  =  Position,  Navigation,  and  Timing 

REAP  =  Resilient  Enterprise  Architecture  Pathfinder 

SEAS  =  System  Effectiveness  Analysis  Simulation 

SMC  =  Space  and  Missile  Center 

SOF  =  Special  Operations  Force 
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STK  =  Satellite  Toolkit 

TLE  =  Two  Line  Element 

TPL  =  Tactical  Programming  Language 

U.S.  =  United  States 

USAF  =  United  States  Air  Force 

WMD  =  Weapon  of  Mass  Destruction 
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Appendix  C.  QUAD  CHART 
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Figure  36.  ENS  Quad  Chart 
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